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FOREWORD 


One  of  the  goal”  of  the  Air  Force  Electronic  Systems  Division  is  the 
development  of  design  principles  and  specifications  for  automated 
training  subsystems  which  could  be  built  into  Air  Force  information 
systems  and  used  for  on-the-job  training  of  :-vstem  personnel.  These 
instructional  innovations  are  aimed  towards  providing  the  improved 
coupling  of  training  and  operations  which  is  essential  for  effective 
use  of  information  systems. 

Task  280104,  Computer-Aided  Instruction  Techniques,  under  Project 
2801,  Information  System  Design  Technology,  is  directed  towards  the 
fulfillment  of  the  above  technical  need.  This  report  is  one  in  a  series 
supporting  that  tasx.  It  presents  research  on  new  techniques  for 
formatting  and  sequencing  information  in  training  subsystems.  Although 
the  principles  involved  are  applicable  to  conventional,  hard  copy  infor¬ 
mation  display  such  as  in  military  manuals,  the  research  is  aimed 
towards  specifications  and  future  experimental  applications  in  dynamic, 
generative  information  displays  sequenced  by  computer. 

The  study  was  performed  between  May  1968  and  July  1969  under 
Contract  F19628-68-C-0212  with  Information  Resources,  Incorporated , 
Cambridge,  Massachusetts,  by  Robert  E.  Horn,  principal  investigator, 
Elizabeth  H.  Nicol,  Joel  C.  Kleinman,  and  Michael  G.  Grace.  The 
Air  Force  task  scientist  and  contract  monitor  was  Dr.  Sylvia  R.  Mayer. 


This  report  has  been  reviewed  and  is  approved. 


IVIA  R. 


SYL 


MAYER,  Ph.D. 


Project  Officer,  Project  2801 


WILLIAM  F.  HEISLER,  Colonel,  USAF 
Chief,  Command  Systems  Division 
Directorate  of  Planning  and  Technology 
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ABSTRACT 


Information  mapping  is  a  method  of  organizing  categories  of  information 
and  of  displaying  them  for  both  learning  and  reference  purposes.  The 
method  may  be  applied  to  the  production  of  self- instructional  books 
or  to  the  organization  of  data  bases  for  computer-aided  instruction  and 
reference.  This  report  is  itself  written  in  modified  information  map 
form.  The  procedures  and  rules  for  information  mapping  were  derived 
from  educational  research  and  technology  as  well  as  from  the  communi¬ 
cations  world.  The  emphasis  is  on  formats  to  communicate  quickly  and 
to  facilitate  scanning  and  retrieval.  The  research  and  development  work 
reported  here  deals  with  the  book  form  of  a  twelve-hour  course  on  sets 
and  probability;  significant  achievement  scores  and  favorable  attitude 
results  were  found  in  several  evaluative  series  with  college  students. 
Because  information  maps  are  composed  of  separable  labelled  information 
blocks,  they  can  serve  as  the  data  base  for  computer  systems  where  both 
learning  and  reference  needs  must  be  met.  Preliminary  work  with  simu¬ 
lated  computer  displays  explored  the  flexibility  with  which  a  system 
so  organized  can  respond  with  a  range  of  user  options  and  display 
variations.  Cost  for  instruction  hour  is  competitive  with  that  of  other 
methods,  but  the  method  has  additional  advantages  in  its  versatility 
and  ease  of  updating. 


TA3L6  OF  CONTENTS 


CHAPTER  1.  INTRODUCTION  TO  INFORMATION  MAPPING  1 
CHAPTER  2.  WRITING  INFORMATION  MAP  BOOKS  20 
CHAPTER  3.  INTRODUCTION  TO  THE  EVALUATIVE  STUDIES  41 
CHAPTER  4.  TOE  TUFTS  UNIVERSITY  TRYOUT  SERIES  43 
CHAPTER  5.  THE  HARVARD  UNIVERSITY  SERIES  SI 
CHAPTER  6.  ASSESSMENT  RESEARCH  STUDIES,  SPRING  1969  66 
CHAPTER  7.  OVERVIEW  OF  THE  EVALUATIVE  STUDIES  100 
CHAPTER  8.  INFORMATION  MAPPING  AND  COMPUTERS  103 
CHAPTER  9.  PRODUCTION  COSTS  114 
CHAPTER  10.  SUWARY  OF  INFORMATION  MAPPING  117 
APPENDIX  122 
REFERENCES  128 
BIBLIOGRAPHY  130 


4 


CHAPTER^  INTRODUCTION  TO  INFORMATION  MAPPING 
OVERVIEW  OF  THIS  REPORT 
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Introduction  Information  mapping  is  the  name  given  to  a  method  of  organ¬ 
izing  and  displaying  information  for  learning  and  reference 
purposes.  This  report  describes  the  research  and  develop¬ 
ment  work  that  has  been  done  with  the  method  in  preparing 
self- instructional  books.  It  also  discusses  exploratory 
work  with  simulated  displays  for  computer- assisted  instruc¬ 
tion.  This  report  is  itself  written  in  a  modified  information 
map  form. 


This  We  describe  what  information  mapping  is,  how  it  began,  and 

Chapter  how  it  was  derived  from  learning  research,  educational  tech¬ 

nology,  and  other  fields  of  knowledge. 


The  Next  The  process  of  writing  information  map  books  is  explained  and 

Chapter  illustrated  with  sample  pages.  Some  content  characteristics 

of  a  set  of  information-mapped  materials  are  reported. 


The  Central  In  several  evaluative  studies,  college  students  learned  from 
Chapters  information-mapped  books  under  different  conditions.  We 

report  the  results  here. 


Later  We  will: 

Chapters 

•  show  how  information  mapping  may  be  used  in  compu¬ 
terized  systems  for  training  and  reference, 

•  report  some  exploratory  tryouts  with  the  simulated 
computer  displays, 

•  describe  some  cost  estimates  for  the  preparation  of 
information-mapped  materials, 

summarize  the  present  status  of  information  mapping. 
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OBJECTIVES  OF  INFORMATION  MAPPING 


Introduction  In  the  past  twenty  years,  we  have  seen  a  significant  increase 
in  research  projects  concerned  with  the  man-information  inter¬ 
face.  The  reasons  for  this  scarcely  need  repeating.  We 
have  more  information  to  handle  in  almost  every  job  and 
discipline.  This  information  is  increasingly  complex. 

People  switch  jobs  more  often,  thus  requiring  more  and  speed¬ 
ier  retraining.  Technology  changes;  men  must  learn  to  use 
the  new.  The  information-generating  capabilities  of  the 
computer  have  surpassed  all  predictions. 

Researchers  are  following  many  lines  of  inquiry  in  an  attempt 
to  augment  the  ability  of  human  beings  to  interact  with  their 
new  information  environment.  Hardware  and  software  extend 
in  many  new  and  more  flexible  directions.  Retrieval  special¬ 
ists  are  seeking  new  ways  of  indexing,  abstracting,  sorting, 
storing,  and  retrieving  information.  Computer- driven  display 
units  are  becoming  widely  available.  Time-sharing  is 
enabling  communities  of  workers  to  share  the  same  data  base. 
Psychologists  and  training  specialists  have  given  much  more 
attention  in  recent  years  to  the  practical  problems  of  how 
human  beings  learn.  Enormous  efforts  are  under  way  to  refine 
programmed  instruction  and  computer-aided  instruction  in  a 
larger  attempt  to  produce  an  "instructional  technology." 


Basic  As  one  response  to  the  burgeoning  educational  demands. 

Aims  information  mapping  has  emerged  as  a  system  of  organizing 

data  bases  for  self- instructional  and  reference  purposes. 
Research  and  development  on  information  mapping  have  been 
concerned  with  these  objectives: 

•  to  make  learning  and  reference  work  easier  and 
quicker 

•  to  make  the  preparation  of  learning  and 
reference  materials  easier  and  quicker 

•  to  develop  economical  procedures  for  designing 
and  maintaining  (e.g.  updating)  training  and 
reference  materials 


2 


INFORMATION  MAPPING: 


ITS  SCOPE 


Basic 

Concept 


Books 


I 


Computer 
Data  Base 


% 


Information  mapping  is  a  system  of  principles  for  identifying, 
categorizing,  and  interrelating  the  information  required  for 
learning-reference  purposes. 

The  system  can  be  applied  to  production  of  books  for  self- 
instruction  or  to  the  specification  of  data  bases  for  computer- 
aided  instruction.  Most  of  the  research  and  development  work 
described  in  this  report  was  concerned  with  information-mapped 
books . 


Information  map  books  are  learning  and  reference  materials 
in  which  cat  "ories  of  information  are  consistently  ordered 
on  the  page  and  are  clearly  identified  by  marginal  labels. 

The  arrangement  of  information  blocks  is  dictated  not  only  by 
logical  analysis  and  classification  of  subject-matter  concepts 
but  also  by  analysis  of  the  contingencies  required  for  success¬ 
ful  learning  and  reference  use.  Therefore,  in  addition  to 
basic  content  material,  information  map  books  also  have: 

•  introductory,  overview  and  summary  sequences 

•  diagrams,  charts,  trees 

•  feedback  questions  and  answers  in  close  proximity 
to  material  to  be  learned 

•  self-tests  and  review  questions 

•  tables  of  contents,  alphabetic  indexes  and  local 
indexes  with  connections  to  related  topics 


Through  our  studies  with  a  book  version  of  an  information 
mapped  subject,  it  has  become  clear  that  similar  techniques 
could  effectively  organize  a  data  base  for  computer-assisted 
instruction. 

The  data  base  would  be  composed  of  Reparable  labeled  blocks 
of  information  together  with  their  interconnections.  This 
would  afford  a  flexibility  in  using  only  those  parts  of  the 
system  that  are  required  for  a  particular  nurpose. 

The  flexible  block-identified  data  base  could  be  rearranged 
for: 


initial  learning 

•  for  the  naive  student 

•  for  the  sophisticated  student 
relearning  or  review 
reference  use 
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IN  A  BOOK  FORM,  INFORMATION  MAPS  ARE.  .  . 


Stored  this  way.  .  . 

.  .  .and  displayed  this  way.  . 

.  .  .on  printed  pages 

- - - - >! 

.  .  .on  the  same  pages 

- 

name 


definition 


connections 


name 


definition 


connections 


BUT  WHEN  USED  IN  A  COMPUTER,  THE  SAME  INFORMATION  MAPS.  .  . 


.  .  .are  stored  this  way.  .  . 


.  .  .In  interconnected  networks 
of  electronically  coded 
information  blocks.  .  . 


definition 
formula  \ 
use - -  x.X 


review-—-''' 
question  1 
answer 


connections 
next  related 


example  1 

\  /^.example  2 
name 

M |  example  n 

/{feedback  question  1 
I  answer  1 
/feedback  question  2 
I  answer  2 
feedback  question  3 
answer  3 

prerequisite 


.  .  .and  may  be  displayed  in  any  of 
the  following  ways.  .  . 


.  .  .as  programmed  instruction-like 
sequences.  .  . 


NATE 

DEFINITION 
EXAMPLE  1 

FEEDBACK  QUESTION  1 


FEEDBACK  QUESTION  1 
ANSWER  1 


.  .  .or  as  information  map  reference.  . 


.and  in  several  other  ways  depending 
upon  learner  control  and  dynamic 
display  capabilities. 


VISIBLE  AND  INVISIBLE  FFATUREb  OF  INFORMATION  MAP  BOOKS 


Introduction  Information  maps  for  self-instructional  books  are  conspic¬ 
uous  for  their  physical  features,  the  format  in  which  they 
present  information. 

An  equally  important  aspect  of  such  information  maps,  however, 
is  that  the  content  itself  is  selected  and  organized  according 
to  a  set  of  underlying  principles. 

The  method  of  presentation  and  the  organization  of  content 
may  be  thought  of  as  the  visible  and  Invisible  features  of  a 
mapped  page. 


c 


V 


Visible  The  more  obvious  visible  characteristics  are  these: 

Features 

*  information  is  presented  in  blocks 

*  marginal  labels  identify  the  kind  of  information 
in  each  block 

*  a  consistent  format  is  used  for  each  kind  of  in¬ 
formation:  procedures  follow  one  format,  concept 
maps  follow  another  distinct  format,  and  so  on 

functional  and  uniform  headings  and  subheadings 
are  used  to  make  scanning  easy  and  to  speed  up 
reference  work 

*  each  information  map  begins  on  a  new  page  and, 

in  programs  for  initial  learning,  most  maps  occupy 
single  pages 

feedback  questions  and  answers  are  located  in  close 
proximity  to  the  relevant  information  maps 

*  a  local  index  at  the  bottom  of  maps  provides  page 
nunbers  for  quick  location  of  prerequisite  topics 

CThe  last  two  features  are  not  used  in  technical  reports) 


Invisible  The  arrangement  and  sequencing  of  materials  presented  in 
Features  information  map  formats  are  the  result  of: 

detailed  specification  of  learning  and  reference 
objectives  in  behavioral  terms 

•  specification  of  prerequisites  for  the  subject-matter 
area 


continued  on  next  page 
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VISIBLE  AND  INVISIBLE  FEATURES  OF  INFORMATION  MAP  BOOKS,  continued 


(continued)  *  classification  of  the  subject  natter  into  component 

Invisible  types  (concepts,  procedures,  etc.) 

Features 

*  definition  of  the  contingencies  required  for  success¬ 
ful  learning  and  reference 


ORIGINS  OF  INFORMATION  MAP  FEATURES 


Introduction 


Fields 

Drawn 

Upon 


Coming 

Up 


In  the  effort  to  design  more  efficient  materials  for  learning 
and  reference,  we  drew  upon  accumulated  knowledge  in  science 
and  technology.  Research  findings,  generalizations,  and 
procedures  from  many  areas  were  considered  with  a  view  to 
their  possible  practical  value  for  instruction  or  reference. 


Gradually  we  evolved  the  set  of  guidelines  and  rules  for 
organizing  and  displaying  information  that  we  now  refer  to 
as  information  mapping.  These  guidelines  have  their  origins 
in  such  areas  as  these: 

logical  ant. lyses  of  subject  matters 

*  learning  research  findings 

*  teaching  practice 

•  programmed  instruction  techniques 

•  display  technology 

*  human  factors  research 

•  communications  techniques,  including  effective 
writing  principles 

The  implications  of  the  various  ideas  were  translated  into 
practical  form  and  were  documented  as  rules  or  procedures  for 
preparing  information  maps. 

Example:  A  common  research  finding  is  that  learning 
is  enhanced  when  practice  exercises  and  answers  are 
given  in  close  proximity  to  new  material.  This  finding 
becomes  the  basis  for  the  information  map  rule  that 
a  page  of  feedback  questions  and  answers  should  generally 
be  inserted  after  each  map  of  new  information. 


In  the  next  few  pages,  we  draw  upon  the  field  of  education 
to  illustrate  how  certain  information  map  features  were 
derived.  Ne  also  outline  briefly  the  process  of  designing 
and  developing  learning  materials  in  book  form. 

The  next  chapter  traces  the  actual  process  of  writing  maps 
from  the  present  set  of  guidelines. 
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INFORMATION  MAP  FEATURES  DERIVED  FROM  LEARNING  RESEARCH  AND  TEACHING  PRACTICE 


Introduction  Although  information  map  features  have  their  origins  in 
several  fields ,  there  is  no  doubt  that  their  principal 
foundations  lie  in  education  and  learning  research.  On  the 
chart  below,  we  indicate  briefly  some  of  the  findings  that 
led  to  the  design  of  certain  information  map  features. 

This  chart  (which  is  not  intended  to  be  exhaustive)  is  one 
example  of  the  research  support  behind  information  mapping. 

Naturally,  the  evidence  is  not  all  of  equal  strength ,  but 
we  have  tried  to  bring  to  bear  on  a  practical  task  some  of 
the  most  premising  factors. 

Because  the  experimental  basis  for  some  map  features  is 
extensive,  we  cite  wherever  possible  research  review  articles 
to  put  the  reader  in  touch  with  the  main  sources  of  evidence. 
In  the  citations  below,  such  major  review  articles  are 
marked  by  asterisks  to  distinguish  them  from  reports  of 
original  research . 


These  results  of  educational 
research  lead  to  .  .  . 


Active  responding  generally 
aids  learning. 

(Luasdaine  and  May*,  1965; 
Briggs*, 1968;  Glaser*,  1965) 

Tlie  act  of  writing  responses 
helps  some  learners. 

(Edling*,  1968) 

feedback  or  knowledge  of  results 
(or  ' reinforcement*)  often 
facilitates  learning  by: 

confirming  or  correcting 
learner’s  understanding 
providing  a  motivational 
effect 

*  improving  scanning  be¬ 
havior 

(Luasdaine  and  May*,  1965; 

Smith  ,  1964;  Gagnfc  and  Rohwer*, 
196P;  Glaser*,  1965) 


► 


.  .  .  these  implications  for  the 
design  of  instructional  materials. 


o 


Insert  feedback  questions  after 
introducing  new  materials. 

Locate  answers  conveniently  nearby. 


continued  on  next  page 
8 


V,VW«  *?*’*%*'?*• 


4 


INFORMATION  MAP  FEATURES  DERIVED  FROM  LEARNING  RESEARCH  AND 
TEACHING  PRACTICE,  continued 


i 


•  • 

These  results  of  educational 
research  lead  to  .  .  . 

► 

.  .  .  these  implications  for  the 
design  of  instructional  materials. 

The  insertion  of  questions, 
"test-like  events,"  after  text 
segments  has  a  positive  effect 
on  learning.  Giving  knowledge 
of  results  further  increases 
the  effect. 

(Gagnfc  and  Rohwer*,  1969; 
McKeachie*,  1963) 

0 

Use  feedback  questions  after  maps 
with  new  information  and  use  sets 
of  review  questions  after  natural 
clusters  of  maps  and  at  the  end 
of  topic  treatments.  Provide 
answers  as  well. 

Self-tests,  pretests  facilitate 
retention. 

(Glaser*,  1965;  Briggs*,  1968; 
Bloom*,  1963) 

* 

a 

*  t 

In  concept  learning,  a  variety 
of  examples  promotes  learning. 
(Gagn€  and  Rohwer*,  1969; 
Lumsdaine*,  1963) 

D> 

Use  examples  and  nonexamples  to 
point  up  differences  and  similar¬ 
ities  among  concepts. 

% 

\ 

t' 

*  ;; 

1 

1 

t  1 

Instructions  are  useful  in 
calling  learner's  attention  to 
important  features. 

(Gagnfe  and  Rohwer*,  1969; 

Gagnfe, .1965) 

O 

Use  introductory  paragraphs 
or  previews  to  alert  learner  to 
importance  of  upcoming  ideas. 

V 


4 


Judicious  use  of  underlining 
often  helps  to  focus  attention 
on  key  elements . 

(Hershberger  and  Terry,  1963) 


Underline  important  words  in 
definitions. 


continued  on  next  page 
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INFORMATION  MAP  FEATURES  DERIVED  FROM  LEARNING  RESEARCH  AND 
TEACHING  PRACTICE,  continued 


These  results  of  educational 
research  lead  to  .  .  . 

► 

.  .  .  these  implications  for  the 
design  of  instructional  materials. 

"Cueing"  or  labelling  appears 
to  aid  by  alerting  learner  to 
nature  of  upcoming  information 
and  informing  him  what  his 
learning  task  is. 

(Glaser*,  1965) 

0 

Use  marginal  labels  and 
informative  map  titles. 

Pictorial  materials  often 
help  learning. 

(Briggs*,  1968) 

0 

Use  diagrams  and  drawings  to 
illustrate  concepts  and  procedures. 

For  some  kinds  of  materials , 
charts  of  the  information  are 
valuable. 

(Feldman,  1965) 

0 

Use  tables  and  verbal  matrices 
to  display  concept  relations. 

Simple  sentence  structures  in 

In  general,  use  active  voice 

the  active  voice  make  learning 

0 

and  simple  sentences. 

easier. 

(Gagnt  and  Rohwer*,  1969; 

Coleman,  1965) 
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ORGANIZATION  AND  INTEGRATION  OF  INFORMATION  IMPORTANT  FOR  LEARNING 


Introduction  Some  important  features  of  information  mapping  owe  their 
origins  to  a  topic  of  current  theoretical  interest  among 
learning  psychologists  -  namely,  the  logical  and  psychological 
structures  of  knowledge  and  their  impact  on  learning  and 
retention. 

Theoretical  Piaget  had  long  ago  speculated  that  "learning  ...  is 
Discussions  facilitated  by  presenting  materials  in  a  fashion  amenable 
to  organization"  (Flavell,  1963),  but  it  is  only  in  recent 
years  that  psychologists  have  actively  taken  up  the  problems 
of  how  cognitive  structures  develop  and  of  the  role  of  organ¬ 
ization  in  learning  and  retention. 

The  'atomistic'  approach  of  most  programmed  instruction  ma¬ 
terials  has  been  criticized  (Stafford  and  Conbs,  1967)  and  a 
firm  case  made  for  the  advantages  of  "meaningful  organization 
and  holistic  presentation  of  materials." 

In  a  symposium  on  "Education  and  the  Structure  of  Knowledge" 
(Phi  Delta  Kappa,  1964),  P.H.  Phenix  remarked:  "It  is 
difficult  to  imagine  how  any  effective  learning  could  take 
place  without  regard  for  the  inherent  patterns  of  what  is 
to  be  learned." 

David  Ausubel  (1960,  1963,  1964,  1968)  has  developed  a  logical 
and  psychological  case  for  believing  that  learning  and  long¬ 
term  retention  are  facilitated  by  ’organizers'  which  provide 
an  'ideational  scaffolding.'  He  has  now  amassed  considerable 
experimental  support  for  his  hypotheses. 

The  well-known  studies  of  Katona  (1940)  with  college  students 
pointed  up  the  importance  of  organization  for  learning  and 
for  retention. 

The  relation  of  organization  of  materials  to  ease  of  learning 
also  finds  support  in  the  area  of  verbal  learning  research 
(Underwood*,  1966). 

Implications  Although  many  issues  remain  +*  be  settled  by  research,  a 
for  strong  case  can  be  supported  both  logically  and  empirically 

Information  for  the  advantages  of  organizing  and  integrating  features  in 
Map  Books  materials  for  learning.  Both  verbal  and  graphical  means  can 
be  used  to  inject  a  sense  of  organization  and  direction  into 
a  subject-matter  presentation. 

In  the  practical  effort  to  design  effective  learning  materials, 
we  have  incorporated  a  number  of  features  intended  to  help  the 
learner  integrate  and  organize  the  ideas  for  more  efficient 
storage  in  memory.  These  are  listed  on  the  next  page. 

-  — 


FEATURES  TO  AID  IN  ORGANIZATION  AND  INTEGRATION 


Introduction 


List  of 
Features 


The  following  list  of  features  designed  to  promote  integra¬ 
tion  of  concepts  ar.d  relationships  contains  some  that  we 
have  already  adopted  on  other  grounds.  For  instance;  the 
guidelines  called  for  practice  questions  and  answers  through 
out  the  text  because  learning  research  suggested  their 
value  in  several  ways;  but  questions  can  also  be  phrased 
to  encourage  integration  of  ideas  over  sections  of  learning 
materials . 

Examples  of  maps  showing  some  of  these  features  are  given 
in  Chapter  2. 


*  reviews  and  previews:  to  take  stock  of  the 
ideas  developed  up  to  that  point  and  to  prepare 
the  ground  for  relating  them  to  new  concepts 
about  to  be  encountered 

*  introductions  to  each  map:  to  relate  new  idea  to 
previous  concepts  or  to  familiarize  with  nature 
and  importance  of  new  idea 

'  recaps  or  capsules:  to  summarize  succinctly 
the  essential  ideas  of  rules  or  principles 
in  nutshell  form 

*  tree  diagrams:  to  sketch  the  ideas  and  pro¬ 
cedures  of  a  topic  so  as  to  show  the  role  of 
each  and  its  links  to  others 

*  compare -and -contrast  tables:  to  point  up  the 
similarities  and  differences  between  two  concepts 
that  are  sometimes  confused 

summary  tables:  to  chart  in  easy  reference 
form  the  main  concepts  of  an  area 

review  tests  after  short  sets  of  maps  and  at  the 
end  of  units:  to  promote  the  integration  of 
several  concepts  and  to  practice  using  them  in 
problem  solving 

*  prerequisite  charts:  to  show  schematically  the 
paths  the  learner  can  take  through  a  subject 
matter  in  order  to  reach  the  learning  objectives 


INFORMATION  MAP  FEATURES  FOR  EASE  OF  REFERENCE 


Introduction  In  designing  book-type  materials  for  initial  learning,  we 

added  features  to  facilitate  the  return  to  ideas  previously 
encountered,  an  activity  that  is  often  frustrating  with 
conventional  texts  where  the  contents  of  the  paragraphs  are 
unlabelled.  Common  sense,  human  factors  research,  and 
graphic  technology  were  used  in  formulating  aids  for  easy 
access  to  the  learning  materials.  A  list  of  these  aids 
appears  below. 

It  is  clear  also  that  these  same  features  would  be  important 
for  reference  manuals  or  job  aids.  If  information  map 
materials  were  designed  for  those  purposes  alone,  some  of 
the  introductions,  explanations,  and  examples  needed  for 
initial  learning  would  be  omitted. 

Again  we  n<^e  that  some  of  the  features  needed  for  easy 
reference  purposes  have  already  been  mentioned  as  desirable 
on  other  grounds.  For  example,  labels  on  information  blocks 
aid  in  quick  retrieval  of  ideas  but  they  also  serve  to  alert 
the  learner  to  the  nature  of  his  learning  task  and 
prepare  him  to  take  in  a  specific  kind  of  information. 


List  of 
Features 


V 


continued  on  next  page 


Tables  of  contents  for  learning  books  are  organized 
and  formatted  to  speed  location  of  topics  and  special 
features.  (This  report  does  not  use  the  standard 
format  but  follows  certain  ESD  report  requirements) 

A  predictable  format  for  each  type  of  map 
(concept,  procedure,  etc.)  facilitates  location  of 
needed  information. 

Map  headings  in  consistent  typography  help  in 
scanning  for  page  topic. 

Marginal  labels  help  not  only  in  locating  the  kinds 
of  information  sought  but  also  in  skipping  those 
not  required. 

Local  indexes  at  foot  of  each  map  permit  quick 
location  of  concepts  relevant  to  the  given  map. 

Decision  tables  display  the  choices  appropriate 
for  each  possible  situation. 

Summary  tables  assemble  main  facts  and  relations 
for  easy  review  and  reference. 
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INFORMATION  MAP  FEATURES  FOR  EASE  OF  REFERENCE, 


(continued) 
List  of 
Features 


•  Capsules  provide  "kerne]"  statements  of  key  rules 
or  concepts 

•  Flow  charts  show  graphically  the  sequences  of  events 
in  a  process 

•  Indexes  aid  information  retrieval. 
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OTHER  PARTS  OF  THE  INFORMATION  MAP  SYSTEM 


introduction 


Curriculum 

Planning 


The  Prerequi¬ 
site  Chart 


So  far  we  have  been  concerned  with  what  information  maps 
look  like,  how  they  got  that  way, and  how  they  are  written. 

But  the  process  of  writing  cannot  begin  until  fundamental 
curriculum  plans  are  worked  out.  Furthermore,  the  end  of 
the  writing  task  is  by  no  means  the  end  of  the  production 
process  --  a  crucial  part  of  that  process  is  the  series  of 
try-and- revise  cycles  through  which  the  product  is  refined 
and  the  learning  outcomes  are  brought  closer  to  the  pregram 
objectives. 

The  information  mapping  system,  then,  includes  guidelines  for 
curriculum  planning  and  for  developmental  testing. 


Once  the  subject-matter  area  of  the  project  has  been  agreed 
upon,  a  series  of  interrelated  decisions  must  be  settled, 
including  the  type  of  audience  for  which  the  program  is  in¬ 
tended,  the  conditions  under  which  it  is  to  be  used  and  so 
forth.  When  the  scope  of  the  program  has  thus  been  defined, 
charts  showing  the  nature  of  the  writing  task  are  evolved 
through  the  following  steps: 

•  The  nature  of  the  subject  matter  is  explored  and  the 
potential  topics  are  listed. 

•  The  learning  objectives  for  the  specific  program 
are  determined  and  are  stated  in  behavioral  terms. 

•  The  topics- that  are  required  to  meet  the  specified 
learning  objectives  are  organized  into  a  schematic 
display  called  the  "preliminary  prerequisite  chart"  -- 
a  chart  working  backward  from  the  objectives  to  the 
topics  that  are  required  to  meet  those  objectives. 

•  Analyze  the  nature  of  the  learning  tasks  and  plan 
the  teaching  strategies  for  achieving  them. 

•  Revise  the  prerequisite  chart  to  show  the  assembling 
of  concepts  into  the  networks  of  associations 
building  toward  the  final  instructional  goals. 


This  chart  of  the  topics  and  their  sequencing  plus  special 
learning  materials  serves  as  a  guide  to  the  writer  in  his 
task.  The  process  of  writing  is  illustrated  in  Chapter  2. 


continued  on  next  page 


OTHER  PARTS  OF  THE  INFORMATION  MAP  SYSTEM,  continued 


Successive 

Approxima¬ 

tions 


The  most  important  aspect  of  these  tryouts  is  that  the 
feedback  questions  and  sets  of  review  questions  spaced 
throughout  the  program  give  us  immediate  evidence  of  the  topics 
that  need  amendment  or  expansion. 

Developmental  tryouts  and  revisions  are  key  tools  in  the 
production  of  effective  information-mapped  materials. 


Because  teaching  and  writing  are  both  arts,  we  do  not  expect 
the  first  draft  of  a  learning  program  to  be  totally  success¬ 
ful.  We  rely  heavily  on  the  iterative  process  --  cycles  of 
tryouts  with  students  and  revisions  of  the  materials  in  re¬ 
sponse  to  their  reactions. 


THE  PROCESS  OF  INFORMATION  MAP  PRODUCTION 


MANAGEMENT  DECISIONS 
AND  PLANS 


What  is  the  subject  matter 
and  general  scope  of  the 
project? 

What  general  schedule 
is  required? 

What  general  funding 
scale  is  contemplated? 

What  persons  assigned 
to  job? 


#4  START  j 


CURRICULUM  DECISIONS 
AND  PLANS 


General  Performance 
Specifications,  including 
.  use 
.  users 

.  conditions  of  use 
.  objectives  of  use 

Specify  the  learning  ob¬ 
jectives  in  behavioral 
terms 

Analyze  the  nature  of  each 
learning  task  and  the 
strategy  for  achieving  it 

Prepare  preliminary  Pre¬ 
requisite  Chart,  showing 
topic  sequences  and 
relations 


I 


WRITE  INFORMATION  BLOCKS 


Use  rules  and  formats 
provided  in  Information 
Mapping  Policies 
book. 


PACKAGE  INFORMATION 


as  book,  article, 
final  report, 
etc. 


USE  INFORMATION 
FROM  DATA  BASE 


presented  on 
display  scope 

parts  printed 
in  hard  copy 


TYPE  FINAL  MANUSCRIPT 
OR  ENTER  INTO  DATA  BASE 


DEVELOPMENTAL  TESTING 

Try  out  sequences  of  in¬ 
formation  blocks  to  test 
communication. 


A 


Revise 
where  indicated 


COPY  EDIT 

Check  for  style,  grammar, 
punctuation,  readability, 
format . 


SUBJECT  MATTER/PROCEDURAL  EDIT 


Review  accuracy  of  subject* 
matter  content  and  adherence 
to  information  mapping  procedures. 
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THE  CONTINUING  EVOLUTION  OF  THE  INFORMATION  MAP  SYSTEM 


Introduction 


Developmental 

Testing 


Subject- 

Matter 

Experience 


We  have  mentioned  how  the  guidelines  and  processes  of 
the  information  map  system  first  came  to  be  formulated.  But 
the  initial  statements  were  only  the  beginning  of  a  develop¬ 
ment  process  that  continues  into  the  present. 


The  rules  and  guidelines  were  tried  out  in  the  preparation  of 
learning  materials  in  several  subject-matter  areas.  As 
these  products  took  shape ,  they  were  subjected  to  tryout* 
and-revise  cycles  with  college-age  subjects.  The  students' 
responses  to  feedback  questions  throughout  a  given  map  series 
gave  us  a  basis  for  continuing  improvement  of  the  learning 
units.  But  more  important  in  the  early  stages  was  the 
value  the  responses  had  for  refining  the  system  itself. 

Rules  were  amended,  format  policies  were  changed,  new  pro¬ 
cedures  were  introduced.  The  system  continues  to  evolve 
gradually  as  our  experience  grows  and  as  new  situations  are 
encountered. 


So  far  we  have  applied  the  system  mainly  to  topics  in  math¬ 
ematics.  The  major  part  of  our  experience  was  gained  in 
writing,  testing,  and  reviewing  a  150  page  introduction  to 
sets  and  probability.  This  book  of  mapped  learning  materials 
constitutes  a  ten  hour  self- instructional  course;  it  served 
as  the  main  research  vehicle  for  the  studies  reported  in 
Chapters  3  to  7. 

It  was  also  an  important  influence  on  the  development  of  the 
system  itself.  Maps  from  this  work  are  shown  in  Chapter  2 
to  illustrate  the  writing  process  and  to  show  the  nature  of 
the  learning  materials. 

Other  subject  areas  with  which  the  system  has  been  tried  are: 


SUBJECT  MATTER 

APPROX.  NO. 
OF  INFOMAPS 

Computer  programming 

75 

The  binary  number  system 

60 

Convrs,  an  experimental  computei 

language 

150 

Canard,  a  simulation  language 

150 

Introduction  to  descriptive  statistics 

75 

Introduction  to  matrix  algebra 

35 

Permutations,  combinations,  and 

the  binomial  theorem 

50 

continued  on  next  page 
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THE  CONTINUING  EVOLUTION  OF  THE  INFORMATION  MAP  SYSTEM,  continued 


(continued) 

Subject- 

Matter 

Experience 


Comment 


A  wider  range  of  topics  was  explored  by  a  group  of  graduate 
students  in  a  summer  school  course  in  1967.  They  prepared 
brief  units  on: 

•  basic  concepts  of  operant  conditioning 
some  topics  in  American  history 

•  a  variety  o*  educational  research  concepts  and 
procedures 

two  dentistry  topics:  how  to  extract  a  tooth,  and 
periodontology 

a  topic  in  chemistry:  the  structure  of  the  atom 

•  the  Munsel  color  system  in  art 

•  darkroom  procedure  in  photography 

•  several  topics  in  mathematics 


Experience  with  other  subject  areas  will  undoubtedly  raise 
new  classification  and  display  problems  for  which  guidelines 
will  have  to  be  devised.  But  we  expect  that  the  main 
impetus  to  the  evolution  of  the  system  will  come  from 
continuing  research. 


CHAPTER  Em  WRITING  INFORMATION  MAP  BOOKS 


Introduction  To  show  how  the  rules  and  policies  guide  the  writing  of  infor¬ 
mation  maps  and  to  give  the  reader  some  experience  with  actual 
learning  materials,  we  describe  in  this  chapter  some  major 
types  of  maps  and  illustrate  them  with  sample  pages.  These 
are  taken  mainly  from  the  book  on  sets  and  probability 
that  we  used  in  the  research  to  be  reported  in  later  chapters. 

Not  all  of  the  various  types  of  information  maps  can  be  illus¬ 
trated  by  this  subject  matter.  For  example,  there  would  be 
no  need  in  this  material  to  use  a  process  map,  one  that  shows 
a  structure  changing  over  time. 

Map  A  full  listing  of  the  present  classifications  of  map  types  with 

Classification  the  kinds  of  information  blocks  that  may  appear  on  each  is 
Chart  presented  in  the  appendix.  Because  information  mapping  is  a 

growing,  changing  system,  we  do  not  regard  these  lists  as 
being  fixed  or  complete.  It  is  probable  that  deeper  explor¬ 
ations  into  quite  different  subject-matter  areas  would 
reveal  the  need  for  other  map  types. 

The  This  chapter  describes  the  current  state  of  the  guidelines 

Guidelines  for  writing  some  of  the  common  types  of  information  maps. 

Many  of  these  are  necessarily  general  in  phrasing.  And  ob¬ 
viously,  no  matter  how  specific  they  might  become,  they  can 
never  eliminate  the  need  for  competent  writers.  Many  of  the 
more  important  maps  (such  as  reviews,  previews,  and  summaries) 
require  a  certain  degree  of  "artistry"  in  order  to  be  properly 
effective . 

The  task  of  working  up  interesting  examples  and  feedback 
questions  is  especially  burdensome  and  demanding  because  in¬ 
formation  map  books  use  so  many  of  these.  The  skill  with 
which  this  task  is  done  makes  all  the  difference  between  an 
amusing, challenging  book  and  a  dull,  boring  one. 

This  In  our  descriptions  on  the  following  pages,  we  assume  that 

Chapter  decisions  have  already  been  made  about  the  major  curriculum 

issues:  the  scope  of  the  project,  the  nature  of  the  intended 
audience  and  specification  of  the  desired  learning  outcomes. 

In  this  chapter  we  illustrate  the  nature  of  the  writing 
task  from  that  point  on. 

The  chapter  ends  with  an  account  of  some  of  the  content 
characteristics  of  a  completed  set  of  information  maps. 


continued  on  next  page 
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Note  on  The  sample  pages  presented  in  this  chapter  are  maps  that  take 

%  the  Unit  up  only  single  8  1/2  x  11  inch  pages.  This  arises  from  the 

Map  fact  that  the  authors  made  a  conscious  effort  to  limit  each 

map  to  the  single  page.  There  were  certain  practical  and 
*  psychological  advantages  in  this  "modular'*  approach.  However, 

in  certain  cases  of  difficult  topics  or  long  procedures,  it 
occasionally  became  necessary  to  depart  from  that  format. 

In  such  instances,  the  map  might  take  up  two  pages \  we  have 
nc  cases  requiring  more  than  two.  A  map  always  begins  on 
a  new  page. 


PREPARING  THE  PREREQUISITE  CHART 


Introduction 


Definition 


Preliminary 

Step 


Example 


An  important  tool  in  specifying  the  topics  that  you  want  to 
present  to  your  audience  is  the  prerequisite  chart.  This 
ser.  is  to  clarify  the  relationships  inherent  in  the  subject 
matter  and  the  teaching  strategy  to  be  used.  Prerequisite 
charts  are  not  entirely  new  on  the  instructional  scene, 
being  occasionally  used  in  curriculum  planning.  (The  term 
prerequisite  must  not  be  taken  to  mean  the  prerequisite  skills 
a  student  should  have  mastered  before  taking  tip  a  certain 
course  of  study.) 


A  prerequisite  chart  is  a  graphical  presentation  of  the 
topics  that  will  be  cc  ered  in  the  information  map  book.  It 
is  called  a  prerequisite  chart  because  in  the  network  format 
the  relation  of  each  concept  to  others  is  immediately  visible  - 
for  any  given  concept  one  can  identify  the  topics  that  are 
prerequisite  for  it.  It  is  intended  to  reflect  not  only  the 
relationships  within  the  subject  matter  but  also  the  teaching 
strategy  to  be  used  in  presenting  the  topics.  (An  example 
is  shown  on  page  26.) 


The  first  step  in  writing  an  information  map  book  is  to  list 
all  the  topics  that  might  reasonably  be  included  in  a  course 
on  this  subject  at  the  appropriate  level.  The  point  to  keep 
in  mind  is  that  the  topics  listed  should  not  cover  the 
entire  breadth  of  knowledge  contained  in  the  subject,  but 
only  those  topics  that  are  appropriate  for  the  intended  user 
population.  This  list  is  subject  to  revision  as  developmental 
testing  proceeds. 


Our  unit  on  sets  was  written  for  college  students  in  the 
behavioral  sciences,  and  was  intended  to  serve  as 
background  material  for  a  unit  on  elementary  probability. 
The  topics  we  thought  should  be  included  were: 

concept  of  a  set 
elements  of  a  set 

*  describing  sets 
Venn  diagrams 
subsets 

*  the  number  of  subsets  in  a  set 

*  null  set 

*  universal  set 
complement 

*  union 

*  intersection 

*  difference 


continued  on  next  page 
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PREPARING  THE  PREREQUISITE  CHART,  continued 


(continued) 

Example 


Planning  the 
Sequence 


Comment 


•  disjoint  sets 

*  finite  and  infinite  sets 

Since  the  purpose  of  the  unit  was  to  give  those  aspects  of 
set  theory  that  would  be  helpful  in  presenting  the  elementary 
theory  of  probability  to  students  with  a  minimal  background 
in  mathematics,  we  decided  to  omit  the  distinction  between 
finite  and  infinite  sets  (our  treatment  of  probability  was 
only  to  cover  finite  sample  spaces). 


Because  the  topics  in  the  list  are  obviously  interrelated, 
the  order  in  which  they  are  presented  must  be  planned.  This 
is  where  the  prerequisite  chart  helps. 

Two  things  must  be  kept  in  mind  when  ordering  the  topics. 
First,  there  is  usually  no  one  logical  sequence  inherent  in 
a  subject  matter.  Thus,  it  is  up  to  the  authors  to  decide 
upon  the  logical  sequence  of  the  steps.  This  would  usually 
involve  the  philosophy  and  "esthetic  sense"  of  the  authors 
about  the  stfcject  matter. 

The  second  point  is  that  the  sequence  of  presentation  involves 
a  "teaching  strategy."  The  logical  order  chosen  in  the 
initial  charting  of  the  subject  matter  may  not  be  ideal  from 
a  teaching  point  of  view.  And  so  the  prerequisite  chart 
will  be  adjusted  according  to  the  authors'  perception  of 
the  learners'  needs  and  of  the  most  effective  means  of 
serving  them.  The  plan  may  still  undergo  considerable  change 
after  developmental  testing. 


The  aim  of  this  preliminary  analysis  is  to  get  the  topics 
down  to  single  page  map  size.  For  example,  "operations 
with  sets"  is  a  topic  that  must  be  broken  down  further  since 
we  could  not  cover  all  operations  in  a  single  map  for 
students  at  the  intended  level.  Again,  developmental 
testing  will  indicate  where  topics  need  to  be  extended  ever 
two  or  more  maps. 

The  graphical  presentation  of  the  prerequisite  chart  is  also 
helpful  in  delegating  writing  tasks,  since  presence  of 
nearly  discrete  clusters  in  the  subject  matter  will  be  quite 
evident  and  can  be  taken  into  account  when  assigning  the 
tasks. 
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IDENTIFYING  THE  NEED  FOR  SPECIAL  MATERIALS  FOR  LEARNING  AND  REFERENCE 


Introduction  In  addition  to  subject-matter  topics,  additional  materials 
to  aid  learning  and  review  are  needed  at  many  points.  Some 
of  the  information  map  forms  that  serve  these  purposes 
are:  procedures,  compare- and- contrast  maps,  tables,  reviews, 
previews,  summaries,  and  the  presentation  of  certain  impor¬ 
tant  relations  that  follow  logically  from  previous  pages  but 
which  cannot  be  left  to  the  students  to  discover. 

The  guidelines  for  writing  can  only  indicate  in  a  general 
way  the  kinds  of  places  where  these  should  be  added.  Many 
of  them  will  be  added  after  developmental  testing  has  indi¬ 
cated  some  unanticipated  difficulties. 


MAP  TYPE 

DESCRIPTION 

EXAMPLE 

Procedure 

Procedure  maps  explain  a  se¬ 
quence  of  steps  which  must  be 
followed  in  order  to  obtain  a 
desired  output. 

A  procedure  map  would  be  used  to 
explain  how  to  compute  the 
standard  deviation  of  a  set  of 
numbers . 

Compare- 

and- 

Contrast 

Table 

A  comp are-and-contrast  table 
is  used  to  show  the  rela¬ 
tionship  among  certain 
concepts  that  naturally 
group  together.  It  serves 
to  remove  confusion  and 
create  finer  discrimination 
for  the  learner. 

In  set  theory,  a  compare- an d- 
contrast  table  would  probably 
be  useful  for  "union  and  inter¬ 
section"  and  for  "difference 
and  complement." 

Review, 

Preview, 

and 

Summary 

These  maps  are  presented  before 
or  after  a  group  of  maps  that 
form  a  natural  or  arbitrary 
unit.  They  review  what 
has  been  given  thus  far, 
showing  how  it  fits  together, 
and  how  it  is  related  to 
what  will  follow.  Since  the 
best  format  for  these  maps 
will  vary  with  the  type  of 
information  that  has  pre¬ 
ceded,  no  one  guideline  for 
these  maps  will  be  presented 
but  different  examples  will 
be  given. 

One  "natural"  place  to  review 
and  preview  what  is  to  come  in 
the  set  unit  is  after  the  ele¬ 
mentary  language  of  set  theory 
and  before  the  maps  on  set 
operations .  Another  summary 
would  be  in  order  after  the 
entire  unit. 

continued  on  next  page 
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IDENTIFYING  THE  NEED  FOR  SPECIAL  MATERIALS  FOR  LEARNING  AND  REFERENCE,  continued 


MAP  TYPE 

DESCRIPTION 

EXAMPLE 

Review 

Questions 

In  addition  to  review  pages 
summarizing  the  ideas  presented 
up  to  that  point,  we  also  in¬ 
sert  at  frequent  intervals 
sets  of  questions  requiring 
the  learner  to  integrate 
previous  ideas  and  to  prac¬ 
tice  using  them  in  problem 
solving.  They  are  also  pro¬ 
vided  .  *•  the  end  of  units . 

After  maps  on  set  union  and  in¬ 
tersection  was  a  natural  place 
to  insert  a  set  of  review 
questions  requiring  the  student 
to  practice  identification  in 
various  examples. 

Implica¬ 

tion 

Some  maps  present  information 
that  is  implied  by  previous 
material  but  which  cannot  be 
left  to  the  student  to  dis¬ 
cover.  Theorems  in  math¬ 
ematics  certainly  fall 
into  this  category,  but 
developmental  testing  may. 
point  to  the  need  for  even 
simpler  implications  to 
be  spelled  out. 

The  theorem  stating  that  a  set 
with  N  elements  has  2N  subsets 
would  be  presented  on  a  separate 
map.  However,  we  also  found 
that  some  students  have  diffi¬ 
culty  forming  unions  of  differ¬ 
ent  types  of  sets.  Thus  we 
added  a  map  called  "Forming 
Unions  from  Sets  with  Different 
Memberships." 

Comment  As  a  result  of  analyzing  learning  contingencies,  certain  of 
these  special  maps  will  be  added  to  the  prerequisite  chart 
before  the  writer  begins  his  task. 


i 
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EXAMPLE  OF  A  PREREQUISITE  CHART 


Introduction 


Example 


Fnvlm  ® 
of  Sot 
Theory 


Before  writing  begins,  the  complete  prerequisite  chart  (indu¬ 
ing  the  special  learning  materials)  is  drawn  up.  This  chart 
functions  as  a  road  map  for  the  writer,  showing  him  the  nature 
of  his  task  and  the  character  of  the  terrain  that  lies  ahead. 

But  the  chart  is  not  only  important  as  a  guide  to  the  writer  - 
it  can  also  give  the  learner  a  clear  idea  of  his  learning  task 
and  of  the  direction  the  map  presentations  are  heading.  There¬ 
fore,  prerequisite  charts  are  included  in  information  map  books. 


The  arrows  are  used  to  indicate  the  logical  structure  of  the 
subject  matter,  i.e.,  the  relationships  among  the  concepts. 
The  numbers  refer  to  the  teaching  strategy  or  t’  sequence 
in  which  the  topics  are  presented.  The  specia'  saming  and 
reference  maps  are  shown  in  boxes. 
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WRITING  CONCEPT  MAPS 


Introduction 


Definition 


Format 


Introduction 

Block 


Definition 

Block 


Notation 

Block 


After  preparing  the  prerequisite  chart,  we  are  ready  to 
allocate  writing  tasks  and  begin  writing.  The  most  frequent¬ 
ly  used  type  of  map  is  the  concept  map.  This  page  describes 
the  format  rules  for  this  map  type  and  the  guidelines  for 
writing  some  of  the  information  blocks  that  can  appear  in 
a  concept  map  (a  fuller  listing  of  permissible  blocks  is 
given  in  the  appendix) . 

The  details  on  this  page  may  be  easier  to  follow  if  the  reader 
refers  to  the  example  on  page  30 . 


Concept  maps  are  used  to  introduce  new  terms  or  topics,  and 
to  present  any  information  that  may  be  regarded  as  a  state¬ 
ment,  concept,  or  definition. 


Each  concept  map  starts  with  a  title  that  may  be  a  term  or 
a  sentence.  The  title  is  usually  the  designation  in  the  pre¬ 
requisite  chart. 

The  information  explaining  a  concept  can  be  sorted  into  various 
types,  such  as  introductory  remarks,  definitions,  diagrams, 
notation,  etc.  On  a  concept  map,  each  type  of  information 
is  blocked  off  separately  and  labelled  in  the  margin.  These 
marginal  labels  facilitate  initial  learning  as  well  as 
scanning  and  reference.  A  student  who  prefers  to  see  examples 
before  definitions  can  do  so  easily.  The  labels  used  most 
often  are  explained  below,  but  when  nonstandard  labels 
would  be  more  informative,  they  may  be  used. 


An  introduction  appears  with  most  concept  maps  in  order  to  link 
the  new  concept  to  those  that  came  before  or  to  familiarize 
the  learner  with  aspects  of  the  new  idea.  If  present,  an 
introduction  is  always  the  first  information  block  on  a 
concept  map. 


The  definition  block  obviously  defines  the  concept  being 
introduced.  If  the  concept  is  not  a  new  term,  but  an  impli 
cation  from  previous  material,  the  definition  block  may  not 
be  necessary.  The  term  being  defined  is  always  underlined. 

A  definition  may  be  introduced  anywhere  on  a  concept  map 
since  in  some  cases  it  might  be  more  beneficial  (from  a 
teaching  viewpoint)  after  an  example  or  two. 


If  appropriate,  a  notation  block  presents  any  notation 
commonly  used  for  the  term  being  defined.  It  follows  immediately 
after  the  definition  block. 


continued  on  next  page 
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WRITING  CONCEPT  MAPS,  continued 


Theorem  or  This  block  is  usually  present  on  a  map  that  does  not  introduce 
Generalization  a  new  terra.  A  generalization  block  may  be  used  if  the  authors 
Block  do  not  consider  the  statement  to  be  important  enough  to  merit 

the  word  "theorem." 


Formula 

Block 


Diagram 

Block 


Example 

Block 


A  formula  may  be  used  to  restate  a  definition,  theorem,  or 
generalization. 

All  formul?^  are  first  written  in  symbols  and  then  in  words 
directly  underneath  the  symbols.  For  example,  the  formula  ex¬ 
pressing  the  definition  of  the  mean  of  a  sample  is  written  as: 

x  a  El 

n 

X - ' - s  / ' - — ^ — ■  ■  ■  — - - 

mean  of  the  X's  »  sum  of  the  X's _ 

the  number  of  X's  in  the  sample 

The  formula  block  follows  immediately  after  the  definition,  theorem, 
or  generalization  to  which  it  refers. 


Wherever  appropriate,  a  diagram  illustrating  either  a  defi¬ 
nition  or  generalization  is  presented.  The  diagram  follows 
immediately  after  the  formula  or  notation  block. 


A  number  of  example  blocks,  labelled  sequentially,  can  be 
presented  at  any  point  on  the  map.  The  number  of  examples 
depends  on  the  difficulty  of  the  concept  being  explained. 
Examples  may  need  to  be  added  after  developmental  testing! 
this  is  easy  since  each  block  is  a  separate  unit.  Diagrams 
should  be  used  freely  within  the  example  blocks. 


Non-example  In  some  cases,  a  new  concept  may  cause  confusion  with  soms- 
Block  thing  the  learner  already  knows.  A  non-example  may  antici¬ 

pate  the  difficulty  and  help  to  clarify  it.  For  example, 
students  often  confuse  the  null  set  with  the  set  containing 
the  number  zero  as  its  only  element.  Thus,  on  the  map  intro¬ 
ducing  the  null  set,  a  non-example  block  was  used  to  make 
clear  this  distinction. 


Comment  A  comment  block  may  be  used  to  present  any  additional  informa- 

Blook  tion  that  might  be  helpful,  but  which  cannot  be  sorted  into 

another  category. 


Related  A  local  index  listing  those  topics  needed  to  understand  the 

Pages  present  concept  (along  with  their  pages)  is  presented  at  the 

Block  bottom  of  the  page. 


< 


continued  on  next  page 
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WRITING  CONCEPT  MAPS,  continued 


(continued) 

Related 

Pages 

Block 


Thus,  if  a  student  is  having  difficulty  with  a  concept,  the 
related  pages  may  help  him  isolate  the  difficulty  and  clear 
it  up  by  re -studying  a  previous  topic.  Only  those  topics 
directly  connected  with  the  concept  being  introduced  should 
be  mentioned;  otherwise,  the  list  would  be  prohibitively 
long. 
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WRITING  FEEDBACK  QUESTIONS 


Introduction  An  information  map  book  has  feedback  questions  after  each 
map,  but  treats  them  as  being  optional  for  the  learner. 

Thus,  the  student  can  choose  whether  he  needs  the  feedback 
or  not. 

Definition  Feedback  questions  are  based  on  the  concept  of  the  previous 

page  and  require  a  minimum  amount  of  integration  with  previous 
topics.  The  purpose  of  feedback  questions  is  to  let  the 
student  find  out  whether  he  has  understood  the  concept 
presented. 

Format  Feedback  questions  are  placed  after  almost  every  concept 

map  (as  well  as  after  certain  other  types  of  maps).  They 
are  omitted  only  when  the  term  introduced  on  a  map  is  too 
simple  to  warrant  any  practice.  This  is,  of  course,  a  matter 
of  judgment;  developmental  testing  may  point  to  the  need 
for  additional  feedback  questions. 

Answers  are  given  with  as  much  explanation  as  seems  feasible. 
In  our  research  program,  the  answers  were  placed  at  the 
bottom  of  the  page.  The  students'  reactions  to  this  were 
mixed:  about  half  liked  the  idea  of  not  having  to  look  up 
answers  in  the  back  of  the  book,  but  the  other  half  thought 
having  the  answers  at  the  bottom  of  the  page  was  too  much  of 
a  "temptation"  -  they  tended  to  peek  at  the  answer  before 
trying  to  solve  the  problem  completely. 

The  type  of  questions  varied:  some  were  true-false,  some 
were  multiple,  choice,  and  some  required  constructed  responses. 
In  general,  the  types  of  questions  written  were  those  judged 
appropriate  for  the  learning  task. 
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EXAMPLE  OF  FEEDBACK  QUESTIONS 


{ 


WRITING  PROCEDURE  MAPS 


Introduction  Some  subject  matters,  especially  in  technical  areas, 

have  many  procedures  that  must  be  followed  in  order  to  obtain 
a  certain  result.  A  formula  in  mathematics  is  a  procedure 
since  certain  steps  must  be  followed  in  order  to  obtain  the 
result  (e.g.,  the  area  of  a  triangle). 


Definition  A  procedure  map  presents  the  sequence  of  steos  to  be  followed 
in  order  to  obtain  the  desired  result. 


Format  A  procedure  map  is  always  titled  "How  to..."  and  is  divided 

in  half  vertically.  The  left  side  gives  a  general  description 
of  the  procedure  and  the  right  side  of  the  page  follows 
through  with  an  example.  A  box  at  the  head  of  the  page 
states  what  is  given  on  the  left  and  the  example  on  the 
right.  The  procedure  is  followed  step  by  step  on  the  left 
and  the  example  is  worked  out  on  the  right. 


I 


l 


i 


33 


EXAMPLE  OF  A  PROCEDURE  MAP 


Introduction 


Example 


This  map  shows  the  general  case  and  the  worked  example  side 
by  side. 


HOW  TO  COMPUTE  THE  STANDARD  DEVIATION 


f. 

I 

V 

E 

N 

A  itt  of  n  motors 

*l-<2 . *n 

EXAWLE. 

Find  the  standard  deviation  of  the  fol~ 
lowing  tor  scores: 

16,12, 61,93.77 ,St. 19 ,95, 11. 71 

STEP 

PROCEDURE 

EXAMPLE 

1 

Add  tho  (Ivon  maters  to 
got  tho  sub: 

rxl 

The  sub  of  the  ten  test  scotcs  is 

EXj-  I6.i2s.s71  •  too 

2 

Square  the  sub  and  then 
divide  by  the  mater  of 
esses  to  get  : 

(EXp’/n 

(EXj)a/n  •  (I00)*/10  -  640000/10 

•  64000 

J 

Add  the  squares  of  the 
given  maters  to  form 

“l  .... 

The  na  of  the  squares  of  the  test  scores 
it 

EX,*-  •6,sS2,S‘ • *s71*  •  6S2S4 

4 

Subtract  the  result  of 

Step  2  froa  the  result 
of  $tep  S: 

IXj'  -  (ixp'/n 

EXj*  -  CEXil’/n  ■  6S234  -  64000-  12J4 

S 

Divide  the  result  of  Step 

4  by  the  nvaber  of  cues 
sinus  one  to  get: 

.  EXg*  -  (EXi)'/n 
n  -  l 

S'  •  -  137.1111 

6 

The  standard  deviation  is  the 
square  root  of  the  result  of 
Step  5: 

S-  «rT" 

s  •  sTTTrrnr  •  11.71 
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WRITING  THE  SUPPLEMENTARY  MAPS  FOR  LEARNING  AND  REFERENCE 


Introduction 


Compare - 
and- 

Contrast 

Tables 


Reviews , 
Previews,  and 
Summaries 


Review 

Questions 


As  we  saw  earlier,  a  number  of  maps  give  supplementary 
information  to  help  the  learner  form  finer  discriminations, 
avoid  possible  confusions,  integrate  several  concepts,  or 
review  previous  ideas.  They  can  also  be  previews  to  prepare 
him  for  the  next  learning  sequences.  No  new  information  is 
presented  on  such  maps. 


A  compare-and-contrast  table  displays  two  concepts,  side  by 
side,  with  their  definitions,  notation,  and  appropriate 
diagrams  in  order  to  show  the  student  exactly  where  the  sim¬ 
ilarities  and  differences  lie. 

A  compare-and-contrast  table  is  always  titled  "Comparing  -- 
and  — "  where  the  concepts  being  compared  are  inserted  in 
the  spaces.  The  page  is  divided  in  half  vertically  with  the 
rest  of  the  page  organized  as  in  an  ordinary  concept  map. 

Usually,  the  definition,  notation,  and  diagram  blocks  for 
the  concepts  are  presented  along  with  examples. 


The  format  for  maps  of  this  type  is  flexible.  Certain  parts 
of  the  subject  matter  may  be  suited  to  a  summary  table, 
while  a  more  descriptive  presentation  may  be  better  for  other 
parts.  The  prerequisite  chart  itself  serves  similar  functions 
of  keeping  the  student  oriented  and  aware  of  the  direction 
the  subject  matter  is  heading.  These  maps  are  probably 
easier  to  write  after  all  the  others  have  been  done.  The  pre¬ 
requisite  chart  will  show  certain  places  where  they  will  be 
appropriate.  However,  the  need  for  more  of  these  maps  may  be 
identified  from  the  results  of  developmental  testing. 

Review  questions  are  presented  after  a  number  of  interrelated 
topics  have  been  introduced  -  frequently  before  a  review 
and  preview  page  or  after  a  summary  map.  The  types  of  ques¬ 
tions  were  similar  to  those  in  feedback  questions.  They  are 
designed  to  help  the  student  integrate  the  concepts  or  pro¬ 
cedures  that  have  gone  before  and  to  give  him  practice  in 
working  with  the  ideas. 

Answers  are  given  along  with  as  much  explanation  as  see*"5 
needed.  Page  numbers  are  included  to  help  locate  the  original 
topic  treatment. 

(Since  review  questions  closely  resemble  feedback  questions, 
we  do  not  give  a  sample  page  here.) 
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EXAMPLE  OF  A  REVIEW  AND  PREVIEW  MAP 


Introduction  The  following  map  was  inserted  after  the  elementary  terms 

of  set  theory  were  introduced.  It  served  as  a  quick  capsule 
summary  of  all  the  concepts  presented  up  to  that  point  and 
gave  a  short  rationale  for  introducing  the  operations  with 
sets  that  were  to  come  later. 


Example 
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REVIEW  AND  PREVIEW 


Past 


Present 


Future 


In  learning  a  new  subject  it  often  helps  to  pause  and  retrace 
the  ideas  covered.  In  doing  this  we- can  get  a  clearer  picture 
of  how  the  ideas  fit  together. 

In  set  theory  we  have  now  acquired  soae  basic  notions  about 
the  nature  of  sets: 

First  we  talked  about  what  sets  are  and  how  to  indicate  then: 

.  a  set  is  any  collection  of  things  or  symbols;  it  is 
connonly  designated  by  a  capital  letter; 

.  the  things  in  a  set  are  called  elenents,  and  the  symbol 
c  shows  that  an  itea  belongs  to  a  set; 

•  *°  apaeify  what  elenents  are  to  be  included  in  a  set, 
we  have  two  aethods: 

1.  the  roster  method  in  which  we  list  the  elenents 
of  the  set  within  braces ,  and 

2.  the  rule  method  in  which  we  give  a  description 
or  a  definition  that  covers  all  the  elenents 
we  intend  to  include. 

Next  the  different  kinds  of  sets  were  defined: 

.  the  universal  set,  called  U,  which  is  the  set  of  all 
units  or  things  under  consideration; 

.  the  null  set,  which  is  a  sat  with  no  nenbers  and  which 
we  refer  to  by  writing  f  or  (  ); 

.  the  sublet .  a  set  than  can  be  fomed  from  any  combina¬ 
tion  of  one  or  more  elenents  from  another  set.  Including 

.  aU  th£  elenents  in  the  set ,  so  we  say  "every 
set  is  a  subset  of  itseif,"  and 
.  no  elements  in  the  ret,  so  we  say  "the  null  set 
I?  a  subset  of  every  set." 

Finally,  we  learned  that  to  determine  the  number  of  possible 
subsets  in  a  set,  we  can: 


.  list  all  the  possibilities  and  count  then,  or 
.  use  the  formula,  2^,  which  is  more  convenient  when  the 
number  of  elenents  (N)  in  the  set  is  large. 


Now  that  we  know  the  basic  kinds  of  sets,  we  are  ready  to  start 
building  with  then.  Many  kinds  of  problems  will  require  that 
we  combine  sets  in  different  ways  to  form  new  sets. 


Hie  operations  that  can  be  performed  to  make  new  sets  from  old 
sets  will  be  explained  on  the  following  pages.  They  are  rather 
similar  to  familiar  arithmetic  operations  with  numbers. 
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EXAMPLE  OF  A  SUMMARY  MAP 


Introduction  This  table  seemed  to  tie  up  the  topics  related  to  conditional 
probability.  Readers  who  are  not  familiar  with  the  subject 
matter  can  nevertheless  note  several  important  features  of 
the  example.  The  introduction  underlines  the  importance  of 
the  topic  and  suggests  to  the  student  the  nature  of  his 
learning  task.  In  the  left  column  simple  questions  are 
paired  with  concrete  examples  to  help  the  student  remember 
the  distinctions.  Formulas  are  as  usual  accompanied  by 
verbal  descriptions.  _ 


Example 


POSTVIEW  OF  CONDITIONAL  PROBABILITY 


Introduction  {conditional  probability  is  one  of  the  aost  useful  tools  In  pro- 
ibabtlity  theory.  We  have  seen  ho.  the  equation  can  be 
!  rearranged  to  give  the  uultiplication  theorea  and  the  indepen¬ 
dence  definition,  both  important  concepts. 

!  The  inter- relatedness  of  these  ideas  is  a  great  boon  to  aeaory. 

If  the  conditional  probability  definition  is  thoroughly  under¬ 
stood  and  stored  in  aeaory ,  it  can  serve  as  the  key  to  unlock 
j  recollection  of  how  the  other  formulas  can  be  derived. 

'For  review,  we  re-state  the  definition  of  conditional  probabil- 
]  ity  along  with  the  concepts  derived  froa  it. 

QUESTION 

CONCEPT  AND  FORMULA 

What  is  the  probability 
of  A,  given  that  B  has 
occurreiT? 

Conditional  Probability: 

P(A]»)  . 

(Given  a  red-haired  person, 
what  is  the  probability 
that  he  has  blue  eye  s':] 

the  conditional  the  probability  that  both' 

probability  of  «  events  will  occur  divided 

A,  given  B  by  the  probability  of  B 

What  is  the  probability 
that  both  A  and  C  will 
occur? 

(What  are  the  chances 
of  winning  Olyapic 
aedals  in  both  track 
and  swiaaing7] 

Pjltiplication  Theorea: 

P(Afl»)  •  P(»)-P(A(») 

'the  probability  'the  probability  of  1  tiaes 

that  both  events  •  the  probability  of  A, 
occur  given  B 

Are  the  events 
|A  and  B  independent'* 

(Is  the  event 
flunking  oath ' indepen¬ 
dent  of  the 
event 

llflunking  history7'1  ] 

Independence  Definition: 

Two  events  are  independent  if  and  only  if: 

P(AOB)  •  P(A)  P(K) 

'the  probability’  the  product  of  the' 

that  both  events  »  separate  probabilities, 
occur 

1  ("liven  two  independent 
j events  A  and  1,  what 
i is  the  probability 
that  both  of  then  occur7 

1 (If  loa  and  1  both  roll 
,i  die,  what  is  the  pro¬ 
bability  that  we  both 
|  get  sines7) 

1 _ 

Multiplication  Rule  for  Independent  Fvents: 

(This  is  the  independence  equation  just  above.) 

.  ..  _J 
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CONTENT  CHARACTERISTICS  OF  A  SET  OF  INFORMATION  MAPS 


Introduction 


The  Structure 
of  an 

Information 
Map  Book 


Research 

Possibilities 


The  fact  that  subject-matter  sentences  are  sorted  into  spec¬ 
ified  categories  and  are  presented  in  labelled  information 
blocks  permits  us  to  describe  the  contents  of  our  learning 
materials  with  more  precision  than  is  usually  possible. 

We  can  count  the  number  of  definitions,  examples,  concepts, 
procedures,  diagrams,  and  so  on.  We  can  also  compute  such 
things  as  the  ratio  of  the  number  of  new  information  pages  to 
the  number  of  old  or  redundant  information  pages. 

The  ability  to  make  such  descriptive  statements  opens  up 
some  interesting  research  possibilities. 


The  information  map  book  on  sets  and  probability  that  was 
used  in  the  research  of  Chapter  6  was  analyzed  in  terms  of 
the  number  of  information  blocks  of  various  kinds.  The  results 
are  giver  in  this  table: 


- — 

Number  of  Maps 

Map  Type 

Sets 

Prob. 

Total 

Concept  Maps 

a.  For  new  terms 

17 

27 

44 

b.  For  implications 

2 

4 

6 

Procedure  Maps 

-- 

4 

4 

Compare/Contrast  Maps 

2 

3 

5 

!  Reviews  and  Previews 

3 

c 

Summaries 

3 

5 

8 

Special  Example  pages 

-- 

6 

6 

Feedback  question  pages 

16 

31 

47 

Review  questions  pages 

4 

-1 

12 

Total 

47 

90 

137 

The  modular  format  of  information  maps  presents  a  unique 
advantage  to  the  researcher  in  that  it  permits  him  to  determi.'.e 
the  frequency  with  which  different  kinds  of  information  appear 
in  his  materials.  This  means  that  he  can  manipulate  experi¬ 
mental  variables  in  the  structure  of  his  materials  and  deter¬ 
mine  the  effects  on  the  learning  outcomes.  For  example, 
questions  about  the  optimal  frequency  of  worked  examples  or 
about  the  relative  effectiveness  of  placing  examples  before 
or  after  definitions  can  be  more  easily  researched  because  of 
the  ease  with  which  the  information  blocks  can  be  moved  about. 


continued  on  next  page 
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CONTENT  CHARACTERISTICS  OF  A  SET  OF  INFORMATION  MAPS,  continued 


(continued)  Subject  matters  themr.elves  :an  be  classified  in  terms  of  the 
Research  relative  frequency  of  concepts  or  of  procedures.  Optimal 

Possibilities  learning  conditions  may  turn  out  to  be  rather  different  for 
the  different  types  of  subject  matters. 
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Introduction 


Background 


Our  General 
Approach 


V 


CHAPTER 


INTRODUCTION  TO  THE  EVALUATIVE  STUDIES 


With  the  advent  of  any  new  educational  product,  questions  of 
its  usefulness  and  of  its  attractiveness  to  potential  users 
must  be  dealt  with.  To  answer  such  questions  for  information 
maps  our  approach  has  been  primarily  to  seek  performance  data 
from  subjects  using  mapped  units  under  different  circumstances 
and  to  tap  subjects'  reactions  to  the  materials  through 
questionnaires . 


Our  approach  to  the  evaluation  problem  may  be  summed  up  by 
saying  that  we  consider  that: 

•  "media  comparisons"  are  to  be  avoided, 

•  developmental  testing  with  tryout-and- revise  cycles 
is  a  key  method  for  producing  effective  programs, 

•  pretest-posttest  differences  are  meaningful  evidence 
for  judging  whether  or  not  a  given  instructional  program 
has  achieved  its  objectives. 

In  actual  practice  this  means  that  we  will  evaluate  specific 
information  map  products  against  explicitly  stated  learning 
objectives. 

The  spuriousness  of  "media  comparisons"  which  abound  in  the  lit¬ 
erature  of  programmed  instruction  has  been  frequently  discussed. 

In  1962,  for  instance,  Stolurow  detailed  his  reasons  for  judging 
media  comparisons  inappropriate  and  he  expressed  the  "prediction 
and  firm  hope  .  .  .  that  the  comparative  study  will  become  extinct." 


In  their  1965  review  of  the  educational  field,  Lumsdaine  and  May 
summarize  the  shortcomings  of  media  comparisons  and  they  applaud 
a  decrease  in  "futile  attempts  to  assess  the  over-all  value  of 
media  by  comparisons  with  'conventional'  instruction,  and  a 
corresponding  increase  in  the  proportion  of  studies  which  attempt 
to  manipulate  specifiable  variables." 


The  evaluation  program  addresses  itself  to  two  issues: 

1.  The  practical  one  of  validating  instructional  sets  of  infor¬ 
mation  maps,  and 

2.  The  more  experimental  one  of  investigating  certain  parameters 
of  these  sets  of  information  maps  and  of  determining  their 
influences  on  instructional  outcomes. 


continued  on  next  page 
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(continued)  Attitude  of  the  user  toward  the  new  method  is  recognized  as  being 
Our  General  a  very  important  facet  of  the  evaluation  program.  It  matters 

Approach  little  if  a  program  is  strikingly  effective  in  teaching,  yet 

repels  students  from  further  contact  with  the  subject  matter. 

Many  educators  are  now  turning  to  attitude  and  user-preference 
data  as  being  even  more  informative  than  performance  data  in 
assessing  the  value  of  instructional  techniques.  In  our  evalua¬ 
tive  series  we  use  several  kinds  of  attitude  questionnaires. 

We  also  have  relied  heavily  on  developmental  testing  of  each  in¬ 
structional  unit.  Preliminary  tests  of  the  materials  were  made  as 
a  matter  of  course  to  guide  us  in  revising  and  improving  the  text. 
They  were  also  used  to  perfect  our  isstruction  and  attitude  pack¬ 
ages  for  the  more  formal  tryouts.  Oglyoneof  these  was  systematic 
enough  to  warrant' separate  attention  hero  and  it  is  included  in  the 
description  of  the  final  series. 


The  Evaluative  The  instructional  materials  evaluated  during  this  project  dealt 
Series  with  topics  in  mathematics:  the  first  units  were  entitled  Mathe¬ 

matics  Essential  for  Statistics;  later  units  were  concerned  with 
elementary  probability  theory.  All  were  aimed  primarily  at  college 
students  in  the  behavioral  sciences  and  the  intention  was  to  make 
the  topics  understandable  even  to  students  with  minimal  mathemati¬ 
cal  backgrounds.  The  materials  were  to  be  self-instructional. 

It  is  convenient  to  discuss  the  evaluations  of  information  maps  in 
terms  of  three  main  testing  periods.  These  together  with  a  brief 
designation  of  details  are  given  in  the  table  below.  Each  study 
will  be  described  in  subsequent  chapters. 


Study 

Designation 

Date 

Topics 

Covered 

No.  of 
pages 

Comparative  Conditions 

Tufts 

University 

Sept. 

1968 

permutations, 
combinations , 
binomial 

84 

controlled  laboratory 
study  versus  home 
study. 

Harvard 

University 

Oct. 

1968 

sets, 

permutations , 
combinations, 
binomial 

126 

information  map  versior 
versus  a  derived 
"prose"  version 

Spring 

Series 

June 

1969 

sets, 

elementary 

probability 

151 

study  for  a  closed- 
book  test  versus 
study  for  an  open-book 
test. 
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Introduction 


Basic  Purpose 


Subjects 


General  Plan 


4 


CHAPTER 


THE  TUFTS  UNIVERSITY  TRYOUT  SERIES 


The  first  planned  evaluative  study  of  learning  from  a  set  of 
information  maps  took  place  in  September,  1968.  The  instruc¬ 
tional  unit  was  entitled  Mathematics  Essential  for  Probability 
and  it  consisted  of  pages  of  maps  and  of  feedback  questions 
covering  the  topics  of  permutations,  combinations,  and  the 
binomial  theorem. 

This  first  tryout  was  a  simple,  short-term  one  designed  mainly 
to  explore  the  effects  of  the  information  map  unit  when  used 
both  under  controlled  laboratory  conditions "and  in  a  "natural" 
mode  of  use  in  home  study. 


The  Tufts  series  was  conducted  to  enable  us  to: 

.  estimate  student  achievement  as  a  result  of  using  the 
learning  materials  under  two  conditions: 

.  in  a  supervised,  timed  study  period, 

.  in  home  study  over  a  week's  time; 

.  discover  students'  assessment  of  the  Information  Map  as  a 
communication  technique  and  of  its  value  in  self-instruction 

.  locate  areas  of  the  text  that  required  adjustment; 

.  obtain  estimates  of  the  time  required  to  study  pages  in 
information  map  form. 


The  22  students  for  this  study  were  members  of  a  course  in 
statistics  for  the  behavioral  sciences  given  in  the  Psychology 
Department  of  Tufts  University.  Eight  students  were  graduate 
students  in  sociology,  ten  were  psychology  majors,  three  were 
joint  psychology-mathematics  majors,  and  one  was  a  mathenic  s 
maj or. 

Typically,  the  class  met  once  a  week  for  three  hours.  Our 
tryout  was  carried  out  during  the  second  and  third  meetings 
of  the  fall  term;  no  materials  duplicating  the  information  map 
topics  were  assigned  to  the  class  before  or  during  this  period. 


The  tryout  of  the  information  map  was  planned  to  consist 
first  of  a  supervised  study  period  in  the  students'  classroom 
during  which  they  would  work  through  the  book  at  their  own 
pace  for  approximately  95  minutes.  Pretests  and  posttests 
would  tap  the  extent  of  their  subject-matter  knowledge  and 
would  record  their  attitude  toward  mathematics  courses  in 
general. 


continued  on  next  page 
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THE  T'JFTS  UNIVERSITY  TRYOUT  SERIES,  continued 


(continued)  Following  that  phase  of  the  tryout,  the  subjects  were  to  take 

General  the  books  home,  work  on  them  during  the  week  and  return  at  the 

Plan  next  class  period  prepared  to  take  a  final  exam. 

In  point  cf  fact,  the  plan  to  have  students  study  at  home  did 
not  work  well  at  all.  The  reasons  appear  to  be  unrelated  to 
this  project,  but  were  rather  a  consequence  of  the  fact  that 
some  students  had  erroneously  enrolled  in  this  course  for 
which  they  were  unqualified;  they  were  therefore  discouraged 
and  demoralized  during  the  home-study  period.  Ten  of  them 
actually  withdrew  from  the  course  a  few  days  later. 

According  to  their  notes  and  comments  to  us,  the  posttest  given 
after  the  intervening  week  was  for  many  a  retention  test 
rather  than  a  test  of  their  grasp  of  new  material.  The  re¬ 
sults  section  will  reproduce  the  relevant  data. 

Data- 

Collection 
Forms 


2.  A  pre-study  attitude  questionnaire  was  designed  to  record 
the  subjects '  feelings  about  mathematics  courses  and  about 
previous  experiences  with  learning  research. 

3.  A  post-study  attitude  questionnaire  tapped  subjects'  reac¬ 
tions  to  learning  from  the  information  map  materials,  to  the 
presentation  methods  of  the  map  book,  to  specific  format 
features  and  toward  the  possibility  of  further  study  from  such 
materials . 

4.  Personal  information  about  each  subject's  major,  year  in 
college,  previous  college  courses  in  mathematics  was  collected. 

5.  The  instructions  asked  subjects  to  write  criticisms  and 
comments  throughout  the  book  whenever  they  wished. 
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1.  Three  subject-matter  tests  were  used  to  measure  the  sub¬ 
ject's  understanding  of  the  topics  before  and  after  studying 
the  information  map  book  under  the  laboratory  situation  and 
after  a  further  home-study  period. 
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THE  TUFTS  UNIVERSITY  TRYOUT  SERIES,  continued 


Procedural 

Details 


STEP 

PROCEDURE 

1. 

Orientation.  General  explanation  of  the  plan  for  the 
tryout  and  the  range  of  material  to  be  covered  was 
given.  Students  already  familiar  with  the  topic  wore 
encouraged  to  leave. 

2. 

Attitude  questionnaire.  Five  brief  questions  were 

completed. 

3 

Pretest.  This  was  administered  under  conditions  where 
each  subject's  time  was  recorded,  but  no  more  than  ten 
minutes  were  permitted  (nor  in  fact,  needed). 

4 

Study  instructions.  Books  were  distributed  and  instruc- 
tions  were  given  to  answer  feedback  questions  and  to 
write  criticisms  as  they  read.  One  hour  and  thirty- 
five  minutes  were  allowed,  including  time  for  a  short 
break  as  each  individual  desired.  Students  were  asked 
not  to  consult  with  one  another  during  the  study  phase. 

5 

Study  termination.  At  the  allotted  time,  work  was 
stopped  and  students  were  asked  to  record  the  number  of 
the  page  on  which  they  were  working  at  that  time.  This 
record  was  collected. 

6 

Post-study  attitude  questionnaires  were  filled  out 

under  timed  conditions. 

7 

First  posttest  of  amount  learned  was  conducted  under 
timed  conditions  similar  to  those  of  the  pretest:  sub¬ 
jects  were  permitted  up  to  ten  minutes  for  this,  but  if 
they  finished  earlier,  the  experimer.ters  recorded  the 
exact  time. 

8 

Homestudy.  The  class  was  instructed  to  finish  studvinp 
the  book  during  the  coining  week  and  to  record  time 
spent  on  the  log  provided;  they  were  to  prepare  for  a 
further  posttest  a  week  later. 

9 

Second  posttest  was  administered  as  before  after  the 
books  and  time  records  had  been  collected. 

10 

Books  were  returned  to  students  who  reauested  them. 
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THE  TUFTS  UNIVERSITY  TRYOUT  SERIES, 


continued 


Data- 

Analysis 

Methods 


Because  this  is  simple  study  of  learning  outcomes  and 
attitudes  toward  this  kind  of  instructional  materials,  the 
data  analysis  methods  consisted  of  t-tests  of  the  two  sets 
of  gain  scores: 

.  gain  in  score  from  the  pretest  to  the  first  posttest 
(given  during  the  supervised,  timed  study  period)  and 

.  gain  in  score  from  the  pretest  to  the  second  posttest 
(given  a  week  later  after  possible  home  study,  although 
in  fact  many  students  did  not  do  any  home  study) . 

Attitude  data  were  simply  summarized  to  indicate  the  strength 
of  subjects'  opinions  about  various  features  of  the  learning 
materials . 

Comments  recorded  in  the  books  were  compiled  as  a  guide  to 
further  revisions  of  the  material.  They  had  a  practical 
objective  and  will  not  be  recorded  here. 
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RESULTS 


►  Achievement 

Scores 


i 


Scores  on  the  subject-matter  tests  were  stated  in  terms  of 
the  percentage  of  items  successfully  passed.  These  are 
given  in  the  table  below  for  the  total  results  of  this  study 
and  also  for  the  separate  topics  covered  in  the  information 
map  material. 


Posttest- 1  was  the  test  given  at  the  end  of  the  supervised, 
timed  study  period;  posttest-2  was  given  a  week  later  after 
the  possibility  of  home  study. 


Section  of  Book 

Pretest 

(N=22) 

Posttest- 1 
(N=22) 

Posttest-2 

(N-19) 

Factorials 
Permutations  and 

68.2% 

100.0% 

92.1% 

Combinations 

19.3% 

61.3% 

69.3% 

Binomial  Expansion 

50.0% 

31.8% 

56.5% 

TOTAL  RESULTS 

32.6% 

72.7% 

61.6% 

TOTAL  RESULTS  WITH 
BINOMIAL  EXCLUDED 

29.1% 

79.6% 

65.1% 

TTie  total  results  line  of  the  table  represents  a  severe 
test  of  the  learning  materials;  the  reason  for  this  is  that 
when  posttest-1  was  given  after  one  hour  and  thirty-five 
minutes  of  study,  well  over  half  of  the  students  had  not  fin¬ 
ished  reading  the  book.  Even  so,  the  improvement  is  statists 
cally  significant. 

A  t-test  of  the  differences  in  scores  from  pretest  to  post- 
test-1  is  highly  significant  with  a  P<.001  (t^  =  5.96, 
df  =  21).  Posttest-2  scores,  those  obtained  a  week  later, 
are  also  significantly  greater  than  the  pretest  scores.  TTiey 
are,  however,  considerably  smaller  than  those  of  the  first 
posttest.  A  possible  reason  for  the  deci-eased  effect  may  be 
found  in  the  students'  time  logs  which  show  that  a  sizable 
percentage  (50%)  of  them  did  not  study  the  materials  during 
the  week . 

If  we  exclude  the  final  topic,  the  binomial,  which  many 
students  had  not  read,  the  results  are  those  of  the  last  line 
of  the  table.  After  studying  the  book  for  about  an  hour  and 
a  half,  the  subjects  scored  correctly  on  almost  80%  of  the 
questions  on  factorials,  permutations  and  combinations. 


continued  on  next  page 
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'  RESULTS,  continued 


(continued)  Whether  we  evaluate  the  total  results  line  or  the  results 

Achievement  with  the  binomial  excluded,  the  gains  from  pretest  to 

Scores  each  of  the  posttests  are  statistically  significant. 

The  central  point  of  the  achievement  score  evaluation  is  simply 
that  the  learning  materials  have  made  a  significant  impact  on 
the  subjects*  knowledge  of  the  subject-matter  area. 

The  differential  rate  of  success  on  the  separate  topics 
showed  that  more  developmental  work  was  needed  in  certain  areas. 
Subjects'  consents  supplement  this  conclusion. 


Time  During  the  timed,  supervised  study  period  in  the  classroom, 

sibjects  were  allowed  breaks  if  they  wished  to  stroll  about 
outside.  The  experimenters  kept  a  record  of  these  times  and 
subtracted  them  in  figuring  each  subject's  total  study  time. 

If  the  total  time  spent  with  the  learning  materials  is  div¬ 
ided  by  the  total  number  of  pages  the  subjects  covered  in 
the  period,  we  have  a  first  estimate  of  how  long  it  takes  to 
read  information  map  pages.  This  estimate  will  be  somewhat 
atypical  because  here  the  subjects' were  asked  to  answer  all 
the  feedback  questions,  whereas  normally  they  would  be  told 
to  work  only  those  where  they  felt  the  need. 


The  total  number  of  minutes  divided  by  the  total  nimber  of 
pages  completed  gives  1.6  minutes  as  the  estimate  of  mean 
time  for  a  page. 


continued  on  next  page 
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V 


RESULTS, 

Attitude 

Results 


continued 


Often  the  most  important  effects  of  an  instructional  program 
lie  in  its  capacity  to  attract  and  sustain  students'  interest. 
For  this  reason  we  were  particularly  concerned  to  learn  our 
subjects'  reactions  to  the  mapped  material. 

Items  in  the  attitude  surveys  consisted  mainly  in  statements 
to  which  the  subjects  responded  by  encircling  one  of  these: 
strongly  agree,  agree,  undecided  or  neutral,  disagree  or 
strongly  disagree.  In  a  few  cases  a  slightly  different  set 
of  response  terms  was  used. 

Rather  than  reproduce  the  qatstions  verbatim,  we  shall  give 
the  gist  of  them  more  tersely  along  with  the  percentage  of 
subjects  expressing  given  views: 

1.  The  learning  materials  were  rated  easy  or  very  easy 
by  86.4%;  no  one  found  them  difficult. 

2.  Learning  by  this  method  was  rated  effective  or  very 
effective  by  90%;  the  remaining  10%  of  the  subjects  were 
undecided;  no  one  rated  them  ineffective. 

3.  As  to  whether  the  learning  materials  progressed  too  slowly, 
55%  said  no,  36%  said  >es. 

4.  Slightly  over  86%  said  th-'y  would  recommend  the  materials 
to  other  students;  9%  said  they  would  not. 

5.  As  to  whether  it  was  difficult  to  adjust  to  the  new  man¬ 
ner  of  presentation,  95%  responded  no;  5%  were  undecided. 

6.  Asked  whether  there  were  too  many  worked  examples,  64% 
said  no;  23%  said  yes. 

7.  The  feedback  questions  were  found  effective  by  86%; 
the  remaining  subjects  were  undecided. 

8.  Asked  if  the  feedback  questions  were  too  numerous,  95% 
said  no;  5%  were  undecided. 

9.  Just  over  72%  of  the  subjects  felt  they  would  retain 
material  learned  from  information  map  books  better  than 
that  from  standard  textbooks;  the  rest  were  undecided. 


continued  on  next  page 
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RESULTS,  continued 


(continued) 

Attitude 

Results 


10.  In  general  the  few  subjects  who  were  not  enthusiastic 
about  this  way  of  learning  were  those  who  had  taken 
between  six  and  ten  semester  courses  in  college  mathematics. 
These  were  students  who  should  have  followed  our  initial 
request  that  those  well  versed  in  mathematics  not  parti¬ 
cipate  in  this  tryout  since  it  was  aimed  at  those  with  a 
minimal  background  in  mathematics.  The  instructions  here 
weTe  unusal  in  requiring  subjects  to  work  out  all  feed¬ 
back  questions;  several  mathematics  majors  reported  that 
this  was  especially  slow  and  frustrating  to  them. 


Summary  of 

Major 

Findings 


Limitations 
of  the 
Study 


1.  After  studying  information  map  units  on  mathematics, 
a  group  of  22  college  and  graduate  students  showed  a 
highly  significant  gain  (P<.001)  in  achievement  test  scores 
over  pre-study  levels. 

2.  They  expressed  a  highly  favorable  attitude  toward  the 
learning  materials. 

3.  A  large  majority  (90%)  found  the  materials  effective  or 
very  effective,  while  the  remainder  were  undecided. 

4.  Differential  scoring  rates  on  subtopics  ana  comments 
obtained  from  the  subjects  served  as  guidelines  in  further 
refinement  of  the  learning  materials. 

The  gain  scores,  while  highly  significant,  may  be 
biased  by  inequalities  between  the  pretest  oiT'the  one  hand 
and  either  of  the  two  posttests  on  the  other.  Although  we 
tried  to  make  the  tests  equally  difficult,  no  procedures  were 
used  to  control  for  possible  differences.  In  view  of  the 
limited  objectives  of  the  tryout,  more  elaborate  methods  and 
statistical  analyses  would  not  have  been  justified. 
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CHAPTERS  THE  HARVARD  UNIVERSITY  SERIES 
THE  EXPERIMENTAL  PLAN 


Introduction  The  second  formal  tryout  of  information  map  materials  was 

carried  out  at  the  Harvard  Graduate  School  of  Education  with 
students  enrolled  in  an  introductory  statistics  course. 

The  students  studied  a  book  of  our  learning  materials  in  a 
"natural"  setting  --  at  home  where  they  usually  study  rather 
than  in  a  controlled  laboratory  situation.  They  did  however 
take  the  achievement  tests  and  attitude  surveys  under  timed, 
supervised  conditions  in  the  classroom  before  and  after  a 
week's  study  period. 

The  learning  materials  for  this  series  consisted  of  the  same 
counting-methods  units  that  the  Tufts  group  had  used  and  in 
addition  there  was  a  42-page  unit  on  set  theory. 

While  the  tryout  was  aimed  primarily  at  learning  more  about 
students'  behavior  with  and  reactions  toward  information 
mapped  materials,  this  study  was  designed  also  to  explore  the 
effects  of  certain  features  of  the  maps. 


Background  The  Harvard  series  was  designed  as  a  first  step  in  research  to 

investigate  experimentally  the  influences  of  certain  features 
of  information  maps  on  instructional  outcomes. 


The  General 
Approach 


An  earlier  chapter  distinguished  between  certain  visible  and 
invisible  characteristics  of  information  maps.  The  visible 
features  included  certain  obvious  aspects  of  formatting, 
spatial  arrangements  and  labelling,  while  the  invisible 
features  referred  to  those  analyses  and  classifications  of 
the  subject-matter  area  that  determined  the  content  of  the 
information  map  text. 

It  was  the  visible  feature.  ?  the  ma^  nat  we  chose  to 
manipulate  experimentally  in  the  Harvard  series. 


Whatever  may  be  the  functions  of  these  visible  features,  the 
structures  that  display  the  text  materials,  it  is  important 
to  know  whether  or  not  they  facilitate  initial  learning. 

For  this  study  two  sets  of  materials  were  prepared: 

a  set  of  information  maps  prepared  in  the  usual  manner,  and 

a  set  of  "prose"  materials  made  by  simply  removing  the 
visible, structural  features,  such  as  boxes,  labels,  unit 
pages,  and  by  running  the  sentences  along  in  standard  prose 
paragraph  form.  The  prose  version  retained  the  same  sen¬ 
tences,  the  same  feedback  questions  and  the  same  learning 
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THE  EXPERIMENTAL  PLAN,  continued 


Experimental 

Groups 


Basic 

Questions  of 
the  Study 


Test 

Materials 


Subjects 


These  two  sets  of  materials  will  hereafter  be  referred  to  as 
the  IM  or  P  versions.  Each  of  the  two  versions  covering  sets, 
factorials,  permutations,  combinations  and  the  binomial 
was  produced  in  book  form  with  tne  title  Mathematics  Essential 
for  Probability. 

Subjects  were  assigned  to  the  IM  or  P  group  at  random. 

Otherwise  the  two  groups  were  treated  alike,  being  instructed 
and  tested  together  as  one  group. 


To  extend  our  understanding  of  the  value  of  information 

maps,  the  questions  we  asked  were: 

•  To  what  degree  can  we  measure  the  effect  of  the  visible 
features  of  information  maps  on  learning? 

•  Do  the  visible  features  appear  to  contribute  to  user 
acceptance  of  the  learning  materials? 

•  What  are  some  of  the  operating  characteristics  of 
information  maps  when  used  in  a  "real"  setting,  i.e.,  where 
students  use  them  at  home? 


A  pretest  and  a  posttest  were  prepared  to  cover  the  subject- 
matter  objectives;  because  class  time  available  for  such 
testing  was  severely  limited,  the  tests  had  to  be  shorter 
than  we  would  have  liked. 

For  the  same  reason,  the  pre-study  attitude  survey  contained 
only  S  items  to  tap  students'  reactions  to  their  previous 
mathematics  courses.  A  16-item  post-study  attitude  question¬ 
naire  was  used  to  register  the  subjects'  opinions  about  the 
form  and  content  of  the  learning  materials. 

Each  subject  was  also  asked  to  provide  supplementary  infor¬ 
mation  about  their  college  and  high-school  mathematics 
backgrounds,  their  field  of  specialization,  and  any  teaching 
or  tutoring  experience. 


t 

| 

I 

j 
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The  subjects  were  all  graduate  students  in  the  School  of 
Education,  Harvard  University,  who  were  enrolled  in  a  course 
entitled  Introductory  Course  in  Educational  Statistics.  They 
had  backgrounds  ranging  from  recent  college  graduates  to 
teachers  to  administrators  of  large  universities  on  sabbat¬ 
ical. 
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EXPERIMENTAL  PROCEDURE 


Introduction 


Procedural 

Details 


The  idea  of  the  experiment  was  presented  to  the  class  at 
their  first  regular  meeting  of  the  semester;  none  of  the 
topics  covered  in  our  experimental  books  were  discussed  in 
class  or  in  homework  assignments  during  the  period  of  the 
experirent. 


— 

TTEP 

PROCEDURE 

1 

Orientation.  On  September  25,  196S,  we  made  a  brief 
announcement  to  the  class  and  distributed  handouts 
explaining  that  we  were  trying  out  some  new  learning 
materials,  that  we  sougnt,  volur veers,  and  how  the  try¬ 
out  schedule  was  plann*  Students  were  assured  that 

in  no  way  could  this  project  iffect  their  course  grade. 

2 

Instructions.  On  Septut&er  30  the  class  was  stopped 

15  minutes  early  to  enable  us  to  carry  out  the  pretest¬ 
ing  of  those  students  who  wanted  to  volunteer  for  the  j 
study.  Of  the  66  students  present,  52  volunteered. 

They  were  given  brief  instructions  about  the  general 
plan  and  then  were  given  both  written  and  verbal  instruc¬ 
tions  about  the  tests  of  Steps  3  and  4. 

3 

Pre-Attitude  Questionnaire.  2  minutes  wore  allowed  (and 

were  sufficient)  for  answering  the  5  questions. 

4 

Pretest.  Each  subject  took  a  ten-minute  pretest  on 
the  subject  matter. 

5 

Random  Assignment  Process.  While  Steps  3  and'4  were 

being  carried  out,  the  names  of  the  students  were 
entered  on  a  previously  prepared  list  of  random  numbers 
and  an  appropriate  copy  of  the  learning  materials, 
either  IM  or  P,  was  labelled  with  each  student's  name. 
These  were  distributed  at  the  end  of  the  test  phase. 

6 

Study  Instructions.  Students  were  told  that  we  were 
testing  two  versions  of  the  materials  and  that  we  asked 
that  they  not  discuss  them  with  one  another  nor  inspect 
the  other  version  until  after  the  experiment. 

The  subjects  were  asked  to  study  the  material  at  home, 
to  record  the  amount  of  time  and  the  number  of  pages 
Involved  in  each  session,  and  to  write  criticisms  and 
comments  freely  on  the  text  pages.  The  personal  data 
record  form  was  to  be  filled  out  as  well. 

continued  on  next  page 


54 


EXPERIMENTAL  PROCEDURE,  continued 


(continued) 

Procedural 

Details 


STEP 

PROCEDURE 

7. 

Post-study  Attitude  Questionnaire.  On  October  7,  1&68, 

in  the  last  15  minutes  o£  the  regular  class  period, 
we  began  the  posttesting  with  the  16-item  ques¬ 
tionnaire  which  required  about  three  minutes.  The 
learning  materials  were  rcllected. 

3. 

Posttest.  The  ten-minute  posttest  was  given  to  41 
subjects. 

9. 

Followup.  Since  5  of  the  IM  group  and  4  of  the  P  group 
were  not  present  for  posttests,  several  steps  were 
taken  to  determine  whether  these  dropouts  differed  from 
other  students.  Five  of  the  9  responded  to  requests 
for  further  information.  We  were  unable  to  detect 
any  differences  between  the  dropouts  and  the  complete 
siijects. 

10. 

Choice  of  IM  or  P  versions .  Since  we  had  to  retain,  the 

study  copies  of  the  books  with  their  very  useful  com¬ 
ments  and  criticism,  we  invited  the  students  who  wanted 
their  own  personal  copy  of  the  materials  to  pick  them 
up  at  our  offices.  This  gave  us  the  opportunity  to 
inquire  into  user  preferences  through  planned  inter¬ 
views  . 

55 


METHODS  OF  ANALYSIS 


Introduction 


Analysis  of 
Covariance 


Other 

Analyses 


The  primary  question  we  were  interested  in  was  whether 
the  two  versions  of  the  Look  were  associated  with  differences 
in  gains  from  pretest  to  posttest  scores.  While  it  is  common 
in  such  cases  to  evaluate  the  differences  in  gain  scores 
between  the  two  groups ,  we  chose  to  use  the  more  precise 
method  of  analysis  of  covariance. 


There  is  always  the  possibility  that  in  spite  of  the  random 
method  of  subject  assignment,  the  two  groups  might  differ  in 
entering  knowledge  of ‘the  subject  matter.  To  guard  against 
this,  we  carried  out  the  analysis  of  covariance  using  the 
pretest  score  as  covariate. 

In  effect;  what  this  accomplishes  is  to  adjust  the  posttest 
scores  so  that  the  effects  of  any  initial  difference  in 
pretest  scores  between  the  groups  is  eliminated.  (Details 
of  the  method  and  rationale  may  be  found  in  Snedecor  and 
Cochran,  1967.) 


We  were  of  course  also  interested  in  determining  for  the  total 
subject  sample  whether  learning  gains  were  significant  regard¬ 
less  of  the  book  version  studied.  An  ordinary  t^test  for 
paired  comparisons  was  used  for  this. 

In  addition  we  shall  present  brief  summaries  of  the  following 
classes  of  data: 

1  the  amount  of  time  spent  in  studying  the  materials, 

•  attitude  scores  of  the  subjects, 

•  mathematical  backgrounds  of  the  groups, 

•  subjects'  preferences  as  revealed  in  choices  of 
IM  or  P  versions  for  their  personal  libraries, 

•  comments  on  the  books  and  on  learning  experiences  with  them. 

It  will  be  recalled  that  a  major  point  of  the  study  was  to 
obtain  detailed  criticism  of  the  books;  thus  for  our  own  use, 
we  compiled  the  subjects'  comments  as  a  guide  to  identifying 
points  of  common  concern.  Some  of  the  kinds  of  changes  sug¬ 
gested  will  be  mentioned. 


56 


RESULTS 


Achievement 

Scores 


Time  Data 


The  IM  group  contained  20  subjects  and  the  P  group  21, 

For  both  groups  the  gain  scores  from  pretest  to  posttest 
wer-;  statistically  significant.  The  mean  pre-  and  posttest 
scores  are  shown  in  the  following  table. 


Group 

No.  of 
Subjects 

Pretest 

Mean 

Score 

Posttest 

Mean 

Score 

Adjusted 

Posttest 

Mean  Score 

| 

IM 

20 

9.50 

19.10 

19.56 

p 

21 

11.52 

18.10 

17.65 

Our  primary  interest  lies  in  the  comparison  of  learning 
effects  for  the  two  groups.  The  analysis  of  covariance,  it 
will  be  recalled,  afforded  a  method  of  adjusting  the  posttest 
means  for  differences  in  the  initial  proficiency  of  the  two 
groups.  Those  adjusted  means  are  shown  in  the  final  column 
of  the  table  above. 

Although  the  IM  posttest  mean  is  larger  than  that  of  the  P 
group,  the  difference  has  no  statistical  significance,  the  t_ 
of  the  difference  being  only  1.17  (38  d. f. ,  P<.2). 

The  complete  details  of  the  analysis  of  covariance  are  pro¬ 
vided  at  the  end  of  this  chapter. 


Students  were  asked  to  keep  a  time  sheet  on  how  they  used  the 
materials.  It  included  the  following  categories: 


Study 

Time 

Time 

How  did  you  use 

Period 

Number 

Day 

Begun 

Ended 

■ .  i  ■ 

Pages 

the  material? 
Comnents . . , 

18  of  the  IM  group  and  20  of  the  P  group  gave  us  this  infor¬ 
mation.  It  was  tabulated  to  provide  answers  to  several 
questions  about  the  use  of  the  books: 

How  many  of  these  finished  the  book? 

.  75%  of  the  IM  group  finished  or  got  within  tne  i,»st  5  pages 

of  finishing  (these  were  pages  that  contained  no  new  information) . 
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RESULTS,  continued 


(continued) 
Time  Data 


.47.6%  of  the  P  group  finished  or  got  within  the  last  5 
pages  of  finishing. 

How  long  did  the  subjects  study? 

Based  cn  those  subjects  who  finished  or  got  within  the  last 
5  pages: 

the  IM  group  spent  an  average  of  5.21  hours  (N  =  15) 

.  the  P  group  spent  an  average  of  3.87  hours  (N  -  10) 

The  subjects'  times  were  highly  variable  and  the  difference 
between  the  two  groups  was  not  statistically  significant 
(t  =  .644,  23  d.f  ,  P  <.5) . 

There  was,  however,  a  significant  difference  between  the 
mean  pretest  scores  for  the  two  groups  of  complete  subjects: 

IM  P 

Pretest  Mean  8.73  14.00 

No.  of  Subjects  15  10 

t  =  2.22  (P  <.05,  23  d.f.) 

This  may  indicate  that  the  P  group  knew  more  of  the  subject- 
matter  than  the  IM  group  and  so  went  through  the  learning 
materials  more  quickly. 


USER  PREFERENCE  DATA 


Introduction  We  invited  the  participants  to  pick  up  their  own  personal 

copy  of  the  materials  at  our  offices  after  the  experiment  was 
over  (Information  Resources,  Inc.  offices  are  3  1/2  blocks 
from  the  Graduate  School  of  Education,  Harvard). 

The  record  of  their  choices  provides  an  interesting  indication 
of  their  reactions  toward  the  two  books  and  some  insight  into 
their  thinking  about  learning  materials. 

When  they  came,  they  were  given  the  two  versions,  IM  and  P,  and 
asked  to  examine  them  and  decide  which  one  they  wanted.  They 
were  r.ot  permitted  to  take  both.  The  interviewers  insisted 
gently  that  they  really  had  to  examine  both  versions  before 
choosing. 

In  one  instance,  the  interviewer  refused  to  accept  the  choice 
of  an  IM  version  by  a  person  who  said  "Oh,  I'll  take  this  one 
because  I  haven't  used  this  kind."  The  interviewer  insisted 
that  the  person  have  a  reason  based  on  a  preference  for  one 
or  the  other  kind  of  materials. 

We  also  interviewed  the  subjects  about 
.  the  experiment 
.  the  learning  materials 
.  their  learning  habits 

The  interviews  lasted  between  5  and  45  minutes  and  averaged 
about  1/2  hour. 


Data  on 
User 

Preference 


TVelve  students  came  to  our  office;  11  chose  the  IM  version. 
Here  is  the  tabulation: 


Version  Chosen 


Version  IM 

Used  P 


IM  P 


6 

1 

5 

0 

USER  PREFERENCE  DATA,  continued 


Interview 

Data 

Picked  IM 


Our  subjects  made  th°sa  comments  after  they  had  examined  both 
IM  and  F  versions  in  response  to  the  question,  "Why  did  you 
pick  this  version?" 


"The  format  in  boxes  is  more  tidy.  I  like  it  better.  It's 
easier  to  read." 

"The  relationships  are  implied.  Things  are  parallel.  If 
you  have  things  related,  you  can  go  through  the  material 
much  faster.  This  book  is  organized  better  than  an  ordinary 
text.  I  liked  the  examples.  You  knew  not  only  what  it  was, 
but  what  it  wasn't.  You  could  see  the  same  example  in 
different  settings.  You  could  see  just  how  inclusive  an 
example  could  be.  People  were  amazed  to  see  how  complicated 
statistics  was  in  our  text  compared  to  how  simple  it  seemed 
here.  Our  author  can't  write  clearly." 

"This  is  the  one  I  really  want.  I'll  get  more  involved  in 
it...  The  layout  is  better  for  me.  I  can  always  grasp  it 
better  when  it's  in  blocks." 

"Oh,  I  prefer  the  boxed- in  one.  I'll  choose  it." 

"The  text  in  our  course  <;oes  much  too  fast  for  me.  15  pages  to 
cover  what  you  cover  in  105.  But  yours  went  faster  than  our 
text  because  it.  was  easier.  I  didn't  have  to  make  up  the 
steps  in  between." 

"I  especially  liked  leatning  how  to  learn  from  your  materials. 
By  the  way,  I  noticed  that  the  blocked  version  had  a  value  of 
telling  me  just  how  much  I  had  learned. . .  it  also  gave  me  a 
clear  stopping  place.  I  could  feel  comfortable  at  stopping 
at  the  bottom  of  the  page." 

"The  boxes  really  appeal  to  me.  I  like  having  things  categor¬ 
ized  in  that  fashion.  When  I  was  using  the  other  book  (P) 

I  felt  that  I  was  getting  it.  However,  when  I  took  the  post¬ 
test  I  realized  that  I  hadn't.  I'd  like  to  go  through  the 
box  version.  I  think  I  might  be  able  to  get  more  out  of  it. 

In  class,  we  did  the  perms,  and  combs,  the  next  week.  Those 
students  who  didn't  participate  in  the  experiment  were  lost. 
The  rest  of  us  found  it  easy." 
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USER  PREFERENCE  DATA,  conti r.ued 


(continued)  "It1  s  compressed  and  the  organization  is  better.  A  mathematical 
Picked  format.  Yeh!  That's  the  way  to  do  it.  Maybe  this  [P]  is 

IM  organized  just  as  well,  but  flow  charts  are  much  better.  The 

trouble  with  most  texts  and  this  book  [P]  is  that  they're  so 
damn  verbal.  You  should  arrange  things  in  a  matrix  form  -- 
it's  so  logical,  so  mathematical  --  it  makes  more  sense  in 
mathematics  to  present  things  in  this  form." 

"I  am  the  ex-president  of  a  college.  One  of  the  interesting  things 
about  your  book  was  that  1  was  initially  turned  off  because  some 
of  the  pages  resembled  'organization  charts'!  And  I  have  unfavor¬ 
able  associations  with  organization  charts.  But  I  decided  to  try 
the  material  anyway.  My  attitude  changed  abruptly  to  a  favorable 
one,  because  I  began  to  have  confidence  in  the  material.  There 
were  places  where  the  materials  broke  down,  that  is,  they  didn't 
teach  me  what  I  wanted  to  know,  but  I  still  felt  confident  that 
I  could  go  on  and  figure  it  out.  You  never  get  this  feeling  from 
an  ordinary  textbook." 


"I  don't  think  that  they  are  useful ...  those  tree-type  things. 

There  seems  to  be  more  verbiage  cut  out  of  this  one[Pj.  I  think 
I'll  take  it.  There's  a  psychological  factor,  I  guess.... It  looks 
easier  to  get  through." 


Picked  P 


SUPPLEMENTARY  RESULTS 


\ 


Attitude  Rssults  1.  45$  of  the  IM  group  and  08$  of  the  P  group  rated  the  ma¬ 

terials  easy  or  very  easy. 

2.  Learning  by  this  method  was  rated  effective  or  very  effec¬ 
tive  by  80%  of  the  IM  and  73%  of  the  P  group;  the  rest 
were  undecided. 


3.  11%  cf  the  IM  and  31%  of  the  P  group  thought  the  material 
progressed  too  slowly,  while  79%  of  IM  and  47%  of  P  dis¬ 
agreed  with  this  statement. 

4.  90%  of  IM  and  94%  of  P  groups  said  they  would  recommend 
the  material  to  another  student;  the  rest  were  undecided. 

5.  When  asked  whether  the  presentation  was  hard  to  adjust  to, 
85%  of  the  IM  group  and  80%  of  the  P  group  said  no.  Cur¬ 
iously  enough,  5%  of  the  IM  group  and  10%  of  the  P  group 
said  yes. 

6.  Asked  whether  there  were  too  many  examples,  85%  of  IM  said 
no  (the  rest  undecided)  but  only  55%  of  P  said  no,  22% 
said  yes  and  the  other  23%  were  undecided. 

7.  95%  of  IM  thought  feedback  questions  were  effective  (the 
rest  undecided)  while  84%  of  P  thought  they  were  effective 
and  5%  thought  they  weren’t. 

8.  Asked  if  there  were  too  many  feedback  questions,  85%  of 
IM  said  no  (rest  undecided),  83%  of  P  said  no,  and  17%  of 
P  said  yes. 

9.  40%  of  IM  and  57%  of  P  group  thought  they  would  retain 
material  learned  this  way  better  than  by  a  standard  text 
(the  rest  were  undecided). 

10.  70%  of  IM  and  84%  of  P  would  like  to  use  similar  learning 

material  for  other  topics  in  the  course  (only  1  student 
in  each  group  disagreed,  the  rest  were  undecided). 
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SUPPLEMENTARY  RESULTS,  continued 


Other  An  analysis  of  the  personal  background  information  of 

Findings  each  subject  showed  that  the  two  groups  were  comparab'e 

in  terms  of  sex,  high  school  and  college  mathematics 
courses  taken,  undergraduate  degree  patterns,  teaching 
experience,  and  attitude  towards  mathematics  in  general. 

.  Male  subjects  judged  the  learning  materials  more 

favorably  than  their  female  counterparts  in  both  groups. 


Limitations  of  No  special  measures  were  taken  to  control  for  possible  diff- 
the  Data  erences  in  pre-  and  posttests.  Biases  here  could  influence 

the  magnitude  of  the  gain  scores  which  for  both  groups  of 
subjects  were  highly  significant. 
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DISCUSSION 


Interpretation  The  lack  of  a  significant  difference  between  the  gain  scores  of 
of  the  Findings  the  two  groups  suggests  one  or  more  of  the  following: 

•  The  visible  features  of  IM's  may  not  contribute  so 

much  to  initial  learning  as  do  the  content  analysis  or  feed¬ 
back  provisions. 

•  The  experimental  procedure,  constrained  by  the  time  re¬ 
strictions  dictated  by  use  of  this  formal  class,  was  not 
sensitive  enough  for  demonstrating  any  differences  that  may 
exist  between  the  two  versions. 

The  latter  possibility  seems  especially  reasonable  to  us. 

The  visible  features  of  IM  might  be  expected  to  operate  pri¬ 
marily  to  alert  one  to  the  nature  of  the  incoming  information 
and  to  facilitate  scanning  for  new  material  or  for  review.  In 
a  one-time  read- through  of  a  short  text,  perhaps  one  should 
not  expect  effects  of  the  visible  features  to  emerge  very 
strongly.  Rather  it  is  in  situations  requiring  information 
retrieval  and  integration  where  their  advantages  might  show 
up.  A  different  set  of  tasks  in  a  less  time-bound  testing 
situation  should  be  more  sensitive  to  such  effects. 

Students  found  the  visible  features  useful  and  appealing. 

The  time  data  as  well  as  the  interview  and  choice  data 
support  this  generalization. 

•  Students  also  singled  out  specific  pages  for  favorable 
unsolicited  remarks.  These  types  of  pages  drew  such  remarks: 
feedback  questions,  compare -and-contrast  pages,  tables, 
overviews,  summary  pages. 

We  obtained  some  solid  information  for  revision  relevant  to 
constructing  information  maps  for  books: 

•  Put  classification  trees  and  prerequisite  maps  at  the  end 
of  units  rather  than  at  the  beginning  (where  they  tended  to 
scare  more  students  than  they  helped). 

•  Build  in  self-tests  instead  of  objectives  pages  at  the 
beginnings  of  units. 

•  increase  the  number  of  maps  in  some  areas  where  students 
had  difficulties. 
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NOTE  ON  STATISTICAL  DETAILS 


Analysis  of 
Covariance 


In  order  to  compare  the  learning  effects  for  the  two  groups, 
we  used  an  analysis  of  covariance  of  the  posttest  scores  using 
the  pretest  scores  as  covariate,  summarized  on  page  57. 

The  results  of  the  analysis  of  sums  of  squares  and  products 
were: 


Source 

d.f. 

Ex2 

Ey2 

Total 

40 

1405 

593 

1270 

Between  groups 

1 

42 

-20 

9 

Within  groups 

39 

1363 

613 

1261 

Reduction  due  to  regression 

1 

- 

- 

275.7 

Deviations  from  regression 

38 

- 

98S.3 

_ 

Regression  coefficient:  b  =  0.45 
Deviations  mean  square  *  25.9 


The  estimated  standard  error  for  the  difference  of  two  adjusted 
means  is: 

sD  =  fl_  (25.9)  {  1  ♦  42  1.63 

/20  V  1363/ 

t  =  19-56  -  17.65  =  1.17  (38  d.f.,  P  <.2) 

1.63 
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ASSESSMENT  RESEARCH  STUDIES,  SPRING  1969 


INTRODUCTION 


Background  In  order  to  obtain  a  set  of  learning  materials  extensive 

enough  to  serve  as  a  serious  research  vehicle,  we  developed 
a  unit  dealing  with  those  topics  in  elementary  probability 
that  are  basic  to  an  understanding  of  statistics. 

The  subject  matter  was  approached  through  the  concepts  of  set 
theory,  which  formed  the  first  unit  of  the  learning  package. 
(The  materials  on  permutations,  combinations,  and  the  binomial 
theorem  were  set  aside  for  the  time  being  with  the  intention  of 
using  them  later  as  a  unit  preceding  elementary  statistics.) 

The  probability  unit  was  developed  by  the  usual  route  of  try- 
out-and- revision  cycles.  When  these  reached  the  point  of 
dwindling  criticisms  from  learners,  we  designed  a  more  formal 
assessment  of  the  program's  effect. 

Although  the  set  unit  was  included  as  an  integral  part  of  the 
package,  it  was  not  evaluated  in  this  assessment  study  because 
it  had  already  been  evaluated  in  the  Harvard  series,  and 
because  time  pressures  induced  us  to  limit  the  tests  to  the 
new  materials. 


General  The  probability  unit  was  used  in  an  assessment  study  that 

Objectives  would  permit  us  to  describe  some  of  the  effects  that  can  be 

expected  when  the  unit  is  used  for  initial  learning  under  var¬ 
ious  conditions. 

The  assessment  includes  statistical  tests  of  pretest-posttest 
gain  scores. 

The  study  was  planned  to  reveal  how  subjects  with  different 
backgrounds  and  degrees  of  mathematical  proficiency  would 
interact  with  the  learning  materials  under  different  conditions 
of  use.  We  were  interested  in  obtaining  information  not 
only  about  the  changes  in  students'  achievement  scores,  but 
also  about  their  attitudes  toward  various  features  of  the 
learning  units  and  toward  possible  future  study  with  such 
materials. 

The  design  was  one  which  would  permit  us  to  indicate  the 
range  of  effects  that  might  be  expected  from  learners  who 
aspired  to  different  levels  of  proficiency  in  the  subject  area. 
And  finally,  we  wanted  to  obtain  information  about  the  useful¬ 
ness  of  the  materials  for  reference  work  as  opposed  to  use  in 
initial  learning. 
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CERTAIN  RESEARCH  ISSUES 


Introduction  In  planning  experimental  programs,  we  consider  alternative 
approaches  and  decide  upon  a  research  strategy  that  we  be¬ 
lieve  will  yield  the  most  informative  results  under  the 
given  circumstances.  In  the  present  case  we  will  describe 
the  reasoning  behind  certain  decisions  in  the  experimental 
plan. 


The  Issues  Three  particular  questions  concerned  us: 

1.  Objectives  for  studying  the  materials, 

2.  The  relevance  of  time  pressures  in  assessing  performance, 

3.  Test  sensitivity  and  the  problem  of  a  reference  point. 

Each  of  these  will  be  discussed  in  turn. 


Learning  When  we  considered  the  variety  of  purposes  for  which  one 

Objectives  might  engage  in  studying  a  particular  subject  matter,  we 

realized  that  it  is  neither  realistic  nor  helpful  to  potential 
users  to  phrase  an  objective  in  terms  of  what  percentage  of 
students  will  pass  an  examination  at  a  given  level  of  profi¬ 
ciency. 

A  student  might  take  up  a  course  because  he  wants  a  general 
understanding  of  the  major  concepts  of  the  area  and  a  know¬ 
ledge  of  where  to  look  for  details  or  procedures  when  the 
need  arises.  His  way  of  using  a  program  would  differ  from 
that  of  a  student  who  was  tzying  to  achieve  a  thorough 
mastery  of  the  subject  in  order  to  pass  the  Graduate  Record 
Examination . 

It  would  be  most  helpful  to  know  how  each  used  the  materials 
and  how  long  the>  spent  to  achieve  their  given  goal. 

It  is  clear  also  that  in  many  formal  courses  it  is  the  instruc¬ 
tor  who  formulates  the  objectives  —  usually  without  consult¬ 
ing  the  victims.  Whether  or  not  the  two  sets  of  goals  agree 
no  one  seems  to  have  inquired. 

Against  this  background  we  decided  to  try  to  make  the  vali¬ 
dation  testing  more  informative  and  relevant  to  realistic 
learning  situations.  In  brief,  our  plan  called  for  students 
to  use  the  information  map  book  undeT  one  of  two  sets  of 
instructions  that  are  designed  to  simulate  common  learning 
situations : 

where  a  student  uses  the  book  in  a  course  that  requires 
a  thorough  understanding  of  concepts  and  procedures  and 
an  ability  to  work  problems  in  closed-book  examination;  and 


continued  on  next  page 
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CERTAIN  RESEARCH  ISSUES,  continued 


(continued)  •  where  the  student  uses  the  book  to  acquire  a  general 
Learning  knowledge  of  the  area  and  enough  familiarity  with  proce- 

Objectives  dures  to  pass  an  open-book  test. 

It  seemed  to  us  that  performance  figures  from  these  two  types 
of  test,  when  coupled  with  time  data  and  personal  information 
about  the  subject,  would  afford  a  more  realistic  picture  of 
the  effects  of  the  instructional  program. 

In  order  to  get  a  lead  on  the  subjects'  own  goals,  as  dis¬ 
tinguished  from  those  we  set  down  for  them,  we  asked  them 
just  before  the  posttest  to  Tecord  the  grade  they  expected 
to  get  on  the  test.  To  complement  this,  one  of  the  questions 
on  an  attitude  survey  asked  about  the  level  of  proficiency 
they  usually  aspired  to  in  mathematics  courses. 

In  actual  use  in  an  educational  system  it  is  quite  possible 
that  these  learning  materials  would  be  used  in  conjunction 
with  other  methods  of  instruction  —  with  an  instructor  and 
classroom  discussion,  for  example. 

In  the  assessment  of  this  program,  however,  we  were  interested 
primarily  in  the  effects  of  the  materials  when  used  alone. 

This  gives  us  the  strictest  test  of  the  units  and  permits 
us  to  estimate  the  minimum  effects  these  units  are  capable 
of  producing. 

The  Time  Many  measures  of  proficiency  depend  upon  the  amount  produced 

Factor  in  in  a  given  time  period  and  a  premium  is  put  upon  producing 

Proficiency  the  greatest  mrnber  of  responses  in  the  shortest  time.  While 

Measurement  such  a  rating  may  be  desirable  for  asseably-line  performance, 
it  seemed  to  us  that  a  time-based  criterion  was  of  limited 
Televance  to  many  learning  situations. 

Accordingly  we  devised  a  posttest  situation  that  permitted 
the  students'  performance  to  be  measured  both  under  a  condi¬ 
tion  of  time  pressure  and  under  a  condition  of  a  generous 
time  allowance.  The  latter  condition  seems  to  be  more  com¬ 
patible  with  the  objective  of  acquiring  a  general  facility 
with  an  area  and  a  knowledge  of  where  to  find  things  when 
they  are  needed.  Speed  of  production  would  not  be  especially 
important  here. 

In  preparing  for  certain  entrance  examinations  for  college  6? 
graduate  school,  however,  speed  and  accuracy  are  both  sought. 

continued  on  next  page 
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(continued)  To  secure  posttest  performance  scores  relevant  to  situations 
The  Time  both  with  and  without  time  pressure,  we  used  the  following 

Factor  in  method: 

Proficiency 

Measurement  1.  During  the  initial  test  phase,  a  time  limit  of  twenty 
minutes  was  imposed;  subjects  were  asked  to  work  as  quickly 
as  possible.  They  were  required  to  write  with  black  pens. 

2.  At  the  twenty-minute  signal,  green  pens  were  substituted 
for  the  black  pens  and  the  subjects  were  allowed  to  continue 
another  forty  minutes  if  they  liked  or  to  turn  in  their 
papers  when  they  were  satisfied  with  their  effort.  They  were 
free  during  this  period  to  correct  any  black  answers  they 
wished  to  change. 

This  technique  was  used  for  both  the  open-book  and  closed- 
book  test  conditions.  It  was  not  used  in  the  pretests  be¬ 
cause  except  for  two  or  three  cases  the  subjects  knew  too 
little  about  the  topic  to  require  more  than  the  twenty-minute 
allotment  or  they  knew  it  so  well  that  they  finished  well 
within  the  twenty-minute  period. 

In  a  study  such  as  this  where  the  goal  of  assesssing  the  ed¬ 
ucational  value  of  the  material  is  combined  with  the  goal  of 
determining  how  its  effects  vary  with  certain  conditions  of 
use,  the  construction  of  achievement  tests  must  be  considered 
from  several  points  of  view. 

In  the  first  place,  it  is  important  to  have  a  test  that  is 
sensitive  enough  to  pick  up  differences  in  comparative  condi¬ 
tions.  In  our  study,  to  be  able  to  measure  the  results  of 
both  open-  and  closed-book  tests,  for  example,  we  must  have 
a  test  that  permits  students  to  demonstrate  the  full  range 
of  their  proficiency  in  the  easier  open-book  situation.  This 
means  that  the  tests  must  be  constructed  to  include  diffi¬ 
culty  levels  that  we  do  not  expect  our  subjects  to  attain. 

It  must  be  complex  enough  so  that  it  does  not  set  a  ceiling 
on  the  scores  of  superior  students.  Thus,  we  are  not  here 
trying  to  write  a  "criterion- referenced"  test  where  students 
are  expected  to  pass  the  majority  of  items. 

Consequently  the  test  was  designed  to  be  difficult  and  beyond 
the  expected  capacity  of  our  subjects. 

But  with  such  a  test,  we  have  the  problem  of  how  to  judge 
the  subjects'  scores.  We  can  no  longer  judge  them  by  saying 
we  expect  a  certain  percentage  of  the  subjects  to  pass  a 
given  percentage  of  items.  In  order  to  have  a  reference  point 


Test 

Sensitivity 
and  a 
Reference 
Point 
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(continued) 

Test  Sensitivity 

and  a 

Reference 

Point 


with  which  to  compare  their  scores,  we  devised  a  competence 
standard  based  on  the  scores  that  certain  "experts”  made 
on  the  same  tests.  Our  "experts"  consisted  of  6  advanced 
candidates  for  the  Ph.D.  in  Statistics  at  a  local  university. 
Four  of  them  had  completed  all  Ph.D.  course  and  examination 
requirements;  all  of  them  had  had  at  least  one  full-year 
course  in  probability  and  mathematical  statistics.  Their 
mean  scores  on  the  subunits  afford  at  least  a  reasonable 
yardstick  for  speaking  about  the  subjects*  achievements. 

We  shall  be  able  to  make  statements  of  this  form: 
after  8-10  hours  study  over  a  two-week  period,  X%  of  the 
subjects  reached  a  level  equal  to  about  70%  of  the  experts' 
scores,  etc. 


DEVELOPMENTAL  TESTING 


Objectives 


Subjects 


Procedure 


Results 


In  order  to  prepare  for  a  formal  evaluation  of  the  revised 
units  on  Sets  and  Probability,  we  used  a  preliminary  group  of 
8  subjects  to  try  out  ?nd  criticize  the  various  materials  for 
us.  These  latter  included  not  only  the  information  map  units 
but  also  the  achievement  tests,  attitude  questionnaires  Before 
and  after  study  of  the  books,  and  instructions  on  use  of  the 
units. 

The  purposes  of  this  tryout  were  to  debug  the  learning  and 
test  materials,  to  obtain  comments  and  reactions  to  the 
total  package,  and  to  gather  study- time  data  to  aid  in  planning 
the  formal  evaluative  study. 


The  8  subjects  may  be  classified  as  follows: 

•  2  graduate  students.  Harvard  School  of  Education 

•  4  undergraduates,  Harvard  College 

•  1  housewife  who  had  finished  1  year  of  college 

•  1  high  school  senior 

In  terms  of  the  College  Qualification  Test  N  (Form  A),  a  stan¬ 
dard  test  for  college  freshmen,  our  group  of  subjects  was  high 
in  mathematical  ability:  5  of  them  scored  above  the  94%  level; 
the  scores  of  the  remaining  3  were  74%,  60%,  and  24%. 


The  numerical  ability  test  just  mentioned,  the  pretest,  an 
attitude  questionnaire  and  a  personal  information  sheet,  were 
all  filled  out  by  the  subjects  in  their  first  session.  Then 
the  two  units  were  given  to  them  to  work  on  at  home,  keeping 
a  time  record  and  comments  on  the  log  provided.  After 
approximately  two  weeks,  the  subjects  returned  at  a  time  of 
their  choice  and  took  the  posttest  and  further  attitude 
questionnaires.  They  were  also  interviewed  to  get  reactions 
in  addition  to  the  comments  written  in  their  study  books. 


The  achievement  tests  given  before  and  after  study  can  be 
broken  down  into  separate  scores  for  the  sets  and  probability 
units: 


Mean 

Mean 

Pretest  Scores 

Posttest  Score 

Sets 

67.5% 

95.5% 

Probability 

57.1% 

89.8% 

The  attitude  data  showed  chat  these  subjects  reacted  favorably 
to  the  materials:  all  thought  that  learning  by  this  method 
was  effective,  no  one  found  difficulty  in  adjusting  to  the 
style,  6  felt  that  they  would  retain  the  material  better  than 
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(continued) 

Results 


Project 

Decisions 


with  a  standard  text  (the  other  2  felt  there  would  be  no 
difference),  and  7  expressed  an  interest  in  studying  other 
topics  written  in  the  information  map  style. 

The  average  time  for  completion  of  the  text  was  7.23  hours  and 
the  median  time  was  6.5  hours. 


As  a  result  of  these  tryouts,  a  number  of  changes  were  made 
in  the  materials  intended  for  the  evaluative  experiment: 

1.  First  of  all,  the  learning  materials  were  revised  to  meet 
the  comments  of  the  subjects;  the  material  on  conditional 
probability  was  expanded  because  some  experienced  difficulty 
with  the  topic. 

2.  Because  the  various  tests  took  up  more  time  than  we  judged 
desirable,  attempts  were  made  to  shorten  them.  The  prerequisite 
test,  which  had  required  thirty-five  minutes,  was  shortened 

to  include  only  items  especially  relevant  to  probability 
problems  --  that  is,  those  concerned  with  manipulating  fractions 
and  solving  algebraic  expressions.  The  test  was  reduced  to 
twenty  items  and  a  time  limit  of  ten  minutes  was  imposed. 

3.  In  the  achievement  tests,  given  before  and  after  study  of 
the  book,  important  changes  were  made.  Because  the  students 
(with  only  one  exception)  had  made  very  high  marks  on  both 
pretests  and  posttests,  we  added  items,  made  some  questions 
more  complex,  and  deleted  easy  questions  passed  by  most  sub¬ 
jects.  This  kind  of  change  was  required  by  our  need  for  tests 
that  would  be  sensitive  to  differences  in  conditions  of  the 
open-  and  closed-book  tests  and  would  set  no  ceiling  on  the 
showing  of  superior  students. 

4.  Nondiscriminating  questions  in  the  attitude  surveys  were 
omitted  and  some  new  questions  were  inserted. 
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THE  EXPERIMENTAL  PLAN 


Introduction  The  experiment  through  which  we  wanted  to  study  the  effectiveness 
of  the  revised  learning  materials  was  designed  with  certain 
built-in  controls  and  certain  procedures  for  eliminating  spur¬ 
ious  influences. 


Test  To  measure  the  subjects’  knowledge  of  probability  before  and 

Materials  after  reading  the  book,  we  drew  up  two  achievement  tests,  which 
we  shall  call  A  and  B.  These  were  equal  in  length  and  as  equiv¬ 
alent  in  difficulty  as  we  could  make  them.  Copies  of  the  two 
tests  are  given  in  the  appendix. 

To  eliminate  the  possibility  that  there  was  in  fact  some  differ¬ 
ence  between  them,  we  arranged  the  test  plan  so  that  Test  A  ser¬ 
ved  as  pretest  for  half  of  the  subjects  and  as  posttest  for 
the  other  half;  similarily  Test  B  was  the  pretest  for  half  of 
the  subjects  and  the  posttest  for  the  other  half.  This  assign¬ 
ment  was  done  at  random. 

With  achievement  scores  obtained  in  this  way,  any  differences  in 
scores  made  before  and  after  reading  the  book  could  not  be  as¬ 
cribed  to  the  fact  that  one  test  was  more  difficult  than  the 
other. 

We  mentioned  in  the  previous  section  that  as  a  result  of  develop¬ 
mental  testing,  these  achievement  tests  were  made  more  difficult. 
Another  result  was  that  the  test  of  numerical  ability,  which  we 
refer  to  as  a  prerequisite  test,  was  shortened.  The  attitude  sur¬ 
veys  which  were  to  tell  us  what  the  students  thought  of  the 
learning  materials  had  similarly  been  revised. 

Those  students  who  were  given  an  open-book  test  also  filled  out 
a  very  brief  questionnaire  about  their  use  of  the  book  during 
the  test. 

Early  in  this  chapter  we  mentioned  that  one  point  we  were  inter¬ 
ested  in  was  how  the  subjects'  own  goals  for  the  course  may  be 
related  to  the  amount  of  time  they  spent  with  the  materials  and 
to  their  achievement  scores  on  the  posttests.  We  tried  to  get 
an  estimate  of  the  subjects'  own  objectives  in  this  way: 
those  achievement  tests  that  were  slated  to  be  posttests  were 
furnished  with  a  cover  sheet  on  which  subjects  were  asked  to 
record  t.ve  grade  they  expected  to  make  on  the  test.  This  "level 
of  aspiration"  figure,  we  conjectured,  would  give  us  an  es¬ 
timate  of  the  subjects’  gcals  for  the  course. 

The  last  page  of  the  test  booklet  asked  what  grade  they  thought 
they  made.  Each  subject  would  also  fill  out  a  personal  data 
sheet  asking  about  his  educational  history,  field  of  major 
interest,  and  experience  in  the  ares  of  mathematics. 
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Experimental 

Design 


Each  subject  was  assigned  to  one  of  three  groups: 

.  open-book , 

.  closed-book  test  group,  and 
.  control  group . 

A  brief  description  of  each  of  these  will  be  given  before  we 
take  up  the  procedure  in  detail. 

The  Control  group  took  the  prerequisite  test,  filled  out  the 
personal  data  sheet,  and  tnen  took  two  achievement  tests  in  a 
row.  They  did  not  see  the  learning  materials  at  all.  The  pur¬ 
pose  in  having  this  group  was  to  find  out  how  much  information 
might  have  been  gained  from  the  tests  themselves  and  to  provide 
a  standard  by  which  the  increase  in  score  from  pretest  to  post¬ 
test  for  the  other  groups  could  be  judged.  The  achievement 
tests  A  and  B  were  each  used  equally  often  for  pretest  and  post- 
test. 

Thr  Closed  group  filled  out  the  personal  data  sheet,  pre-study 
attitude  questionnaire, . the  prerequisite  test,  and  finally  the 
timed  pretest  in  their  first  session.  They  were  then  given  the 
material  to  take  home  to  study  and  to  record  the  time  "pent  on 
it :  An  instruction  sheet  explained  that  they  would  be  given 
a  closed-book  test  when  they  returned.  The  post-study  attitude 
survey  was  inserted  at  the  end  of  the  book  and  was  to  be  filled 
in  before  the  subject  caae  back  for  the  posttest.  When  subjects 
in  this  group  returned,  they  were  given  a  closed-book  test. 
Approximately  two  weeks  elapsed  between  pTe-  and  posttest. 

The  Open  group  want  through  the  same  routine  as  the  Closed  group 
the  first  session.  Afterwards  in  their  instructions  in  the 
front  of  their  books,  they  were  informed  that  the  learning 
objectives  would  be  evaluated  by  an  open-book  final  test. 

The  tests  were  in  essence  individually  administered,  although 
often  several  students  would  come  together  and  would  be  tested 
in  the  same  room.  But  each  was  given  special  materials,  dic¬ 
tated  by  the  plan,  and  each  was  timed  separately. 

The  intention  was  to  secure  approximately  twenty  subjects  for 
each  group;  in  practice  we  fell  slightly  below  that  number. 

The  group  of  experts,  whose  scores  on  the  achievement  tests  were 
to  serve  as  a  reference  point,  were  given  the  tests  under  con¬ 
ditions  similar  to  the  control  subjects.  Half  of  them  took 
Test  A  first  and  Test  B  second;  the  other  half  received  the 
tests  iu  the  opposite  order.  The  order  for  each  expert  was 
determined  at  random.  They  were  not  given  a  time  limit  but 
only  two  of  them  took  longer  than  20  minutes  on  the  pretest 
and  all  took  less  than  20  minutes  on  the  posttest. 


continued  on  next  page 
74 


THE  EXPERIMENTAL  PLAN ,  continued 


Subjects 


These  learning  materials  were  intended  mainly  for  students  in 
the  behavioral  sciences  who  were  often  lacking  in  mathematical 
background  but  who  needed  to  be  prepared  for  statistics.  Accord¬ 
ingly  we  sought  as  subjects  perrons  primarily  of  college  age  and 
we  imposed  no  requirements  of  previous  mathematical  training 
beyond  elementary  high  school  algebra. 

Because  of  the  time  of  year,  however,  we  expected  shortages  of 
test  subjects.  As  a  result  the  subjects  were  not  assigned  to 
groups  strictly  at  random.  When  a  subject  called  about  the  ex¬ 
periment,  he  was  assigned  at  random  to  a  group  and  given  infor¬ 
mation  about  the  work  required  and  the  pay  for  that  group.  At 
times,  however,  a  subject  would  consider  the  pay  not  worth  the 
trouble  of  coming  down  for  the  tests  (if  he  was  in  a  control 
group),  or  the  work  required  taking  too  much  time  (if  he  was 
in  an  experimental  group).  If  this  happened,  he  was  told  of 
the  other  possibility.  The  assignment  of  the  experimental 
subjects  to  the  Open  and  Closed  group  was  strictly  at  random 
since  there  was  no  difference  in  the  work  required. 

The  18  subjects  who  agreed  to  take  part  in  the  experiment  as 
controls  varied  in  age  and  background: 

.  9  were  college  students  (2  in  social  science,  2  in 

education,  2  in  the  arts,  2  in  natural  sciences,  1  in 
nursing,  1  in  dental  technology,  and  1  uncommitted); 

.  2  were  college  graduates  who  were  now  employed; 

.  2  had  1  year  of  college  and  were  looking  for  jobs; 

.  3  were  graduate  students  (in  psychology,  education,  and 

political  science) 

The  backgrounds  of  the  35  experimental  subjects  were: 

.  20  were  college  students  (5  in  social  sciences,  5  in 

education,  6  in  the  arts,  2  in  biology,  1  in  physics, 
and  1  uncommitted); 

5  were  high  school  graduates  (2  were  employed,  3  were 
about  to  enter  college); 
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THE  EXPERIMENTAL 


(continued) 

Subjects 


PLAN,  continued 


6  were  college  graduates  who  were  now  employed; 

.  4  were  graduate  students  (in  psychology,  economics, 

government,  and  art  history). 

The  prerequisite  and  pretest  scores  for  the  three  groups  were: 

Prerequisite  Pretest 

Control  (N  «  18)  13.22  8.44 

Open  (N  «  17)  13.53  7.35 

Closed  (N  «  18)  13.11  8.89 

The  similarity  of  backgrounds  and  initial  test  scores  for  the 
three  groups  *  seems  to  indicate  that  no  serious  bias  resulted  from  the 
non- random  groups  assignments.  If  there  were  any  differences 
in  initial  test  scores  the  method  of  analysis  used  would 
control  for  this. 

The  group  who  served  as  experts  were  six  graduate  students  in 
Statistics  at  a  local  university.  "Each  had  at  least  one 
full  year  graduate  level  course  in  probability  and  mathemat¬ 
ical  statistics.  Four  of  the  six  had  passed  the  qualifying 
examinations  for  the  Ph.D.  degree. 
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THE  EXPERIMENTAL  PLAN,  continued 


Operational 

Procedure 


STEP 

PROCEDURE 

1 

Recruitment.  Subjects  were  recruited  by  advertising 
at  local  colleges  and  using  acquaintances  of  the  in¬ 
vestigators.  Subjects  in  the  Open  or  Closed  group  were 
paid  $70  and  Control  subjects  $5. 

2 

1 

Randomization.  Sixty-two  numbers  were  assigned  to  the 
groups  (and  to  each  order  of  tests  A  and  B  within  a 
group)  at  random.  When  a  subject  called,  he  received 
the  next  available  nuntoer  on  the  list  and  was  assigned 
to  the  group  corresponding  to  that  number  with  the  ex¬ 
ception  explained  in  the  subject  section. 

3 

Orientation.  When  a  subject  arrived,  we  gave  him  a 
brief  account  of  the  purpose  of  the  experiment.  (Con¬ 
trol  subjects  were  merely  told  that  we  were  testing  the 
effectiveness  of  the  examinations  he  would  be  talcing.) 

4 

Form  and  test  administration.  We  gave  the  subjects  the 

appropriate  forms  and  administered  the  prerequisite  test 
(10  minutes),  the  pretest  (20  minutes)  and  the  pre-att¬ 
itude  questionnaire.  They  were  then  given  the  learning 
material  and  asked  to  follow  the  instructions  printed 
inside,  and  return  for  the  posttest  in  2  weeks  at  a  time 
they  arranged  by  phone. 

5 

Posttest.  When  the  subject  returned  for  the  posttest 
we  checked  to  see  that  his  post-attitude  questionnaire 
was  complete,  and  then  administered  the  posttest  in  two 
stages:  during  the  first  20  minutes,  the  subject  wrote 
with  a  black  pen;  then  a  green  pen  was  substituted  and 
he  was  allowed  to  continue  up  to  40  minutes  more.  If  he 
finished  before  that  (as  many  did),  the  exact  time  he 
used  was  recorded. 

If  the  post-attitude  questionnaire  had  not  been  filled 
out  the  subject  did  so  after  the  posttest.  If  the  sub¬ 
ject  was  in  the  Open  group,  he  also  filled  out  the  ref¬ 
erence  attitude  questionnaire. 

6 

Follow-up.  If  a  subject  did  not  arrange  for  the  post- 
test  in  the  expected  interval,  we  called  him  and  he  came 
within  a  day  or  two.  We  were  not  able  to  locate  two 
subjects,  three  returned  the  material  for  personal  rea¬ 
sons,  and  four  were  dropped  from  the  study  because  of  a 
mistake  in  assignment  of  test  forms  to  them. 
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METHODS  OF  ANALYSIS 


Introduction  In  addition  to  presenting  descriptive  results,  we  shall 

report  a  statistical  analysis  of  the  differences  between  the 
experimental  groups  and  the  control  group  using  the  technique 
of  multiple  analysis  of  covariance.  The  rationale  for  this 
has  been  described  in  the  chapter  on  the  Harvard  series. 


Description  In  the  Harvard  study  we  explained  how  the  analysis  of  covar¬ 

iance  provides  a  method  of  adjusting  posttest  scores  for  diff¬ 
erences  in  the  initial  proficiency  of  the  groups,  and  we 
described  how  we  used  the  pretest  as  the  covariate. 

In  the  present  experiment  the  same  considerations  led  us. to  make 
similar  analysis,  which  is  generally  considered  preferable  to 
t-test  comparisons  of  simple  gain  scores  (see  Experimental  and 
Quasi -Experimental  Designs  for  Research,  Campbell  and  Stanley, 
Rand  McNally,  1963) 

In  the  Harvard  study  only  the  pretest  was  available  as  the  co¬ 
variate  but  in  the  present  series,  we  have  pretests  and  pre¬ 
requisite  tests  as  well.  Thus  we  were  able  to  use  both  of 
these  covariates  in  our  analysis  of  the  posttest  scores.  Such 
a  multiple  analysis  of  covariance  is  described  in  Snedecor 
and  Cochran  (1967).. 
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RESULTS 


Introduction 


Achievement 

Scores 


Instructional  programs  have  many  outcomes.  In  everday  usage 
programs  are  studied  for  different  reasons  by  students  who 
differ  in  background  knowledge,  in  natural  aptitude,  and  in 
study  habits. 

Our  aim  was  to  produce  learning  materials  that  the  student  could 
proceed  on  in  his  own  way  with  a  minimum  of  previous  experience. 
Our  test  of  the  program  simulated  two  common  everyday  situations 
where  the  instructor  imposes  the  objectives  of  a  course  in  terms 
of  an  open  or  a  closed  book  test. 

In  addition  we  recognized  that  students  have  their  own  goaK 
regardless  of  the  demands  of  the  instructors. 

The  things  we  want  to  know  about  a  program  are  the  evidences  of 
learning  and  the  amount  of  time  spent  for  the  different  profi¬ 
ciency  goals,  as  well  as  evidences  of  aversive  or  attractive 
powers  of  the  materials  as  revealed  in  the  students'  comments 
and  responses  to  attitude  probes. 


The  posttests  yielded  two  scores:  those  obtained  in  the  20- 
minute  time  period  and  those  obtained  within  an  hour.  For  both 
of  these  measures  the  means  and  standard  deviations  are  display¬ 
ed  in  the  table  below  for  each  of  the  three  groups  of  subjects: 


TEST 

OPEN  (N=17) 
Mean  S.D. 

CLOSED 

Mean 

(n=is: 

S.D. 

CONTROL  (N=18) 
Mean  S.D,. 

Prerequisite 

13.53 

4.80 

13.11 

4.28 

13.22 

4.97 

Pretest 

7.35 

5.59 

8.89 

6.72 

8.44 

8.57 

Posttest- 1 

Unadjusted 

17.18 

8.30 

17.72 

9.10 

9.33 

7.68 

Adjusted 

17.62 

5.72 

17.41 

5.72 

9.24 

5.72 

Posttest-2 

Unadjusted 

22.41 

10.35 

22.22 

9.20 

— 

- — 

The  primary  analysis,  the  multiple  aralysis  of  covariance,  was 
applied  only  to  the  posttest-1  scores,  using  the  prerequisite  test 
and  pretest  as  covariates.  This  is  the  "strictest"  assessment 
of  the  data,  showing  the  effect  that  occurs  under  the  pressure 
of  limited  test  time.  As  a  result  of  this  analysis,  the  posttest-1 
scores  were  adjusted  as  shown  in  the  table  above. 
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RESULTS,  continued! 


(continued) 

Achievement 

Scores 


Test  Forms 
Compared 


Incomplete 

Subjects 


When  these  adjusted  means  are  compared,  we  find  that  each  of 
the  experimental  groups,  Open  and  Closed,  is  different  from 
the  Control  group  to  a  highly  significant  degree.  The 
note  at  the  end  of  this  chapter  gives  the  statistical  details. 

These  posttest- 1  results  show  that  even  with  the  time  restric¬ 
tions,  the  learning  materials  have  significantly  influenced  the 
students'  understanding  of  probability  problems. 

Since  posttest-2  mean  scores  are  higher  than  those  of 
posttest-1,  they  are  obviously  also  significantly  different 
from  the  pretest  scores. 

For  neither  set  of  posttest  scores  would  the  Open  and  Closed 
groups  be  statistically  different  from  each  other.  There  was 
no  apriori  reason  for  expecting  a  difference. 


In  the  results  cited  above,  the  use  of  Fonns  A  and  B  of  the 
achievement  tests  was  equalized  over  conditions,  as  the  test 
materials  section  explained.  Thus  any  differences  in  the  two 
test  forms  could  not  account  for  the  results  obtained.  In 
other  analyses  where  we  want  to  look  into  the  results  in 
relation  to  subjects'  attitudes,  backgrounds,  and  so  on, 
the  two  test  forms  may  not  happen  to  be  equally  represented. 
Therefore  it  is  interesting  to  inquire  first  whether  in  fact 
the  two  forms  do  differ. 

Pooling  Control,  Closed  and  Open  groups ,  we  find  the  following 
scoring  rates  on  the  two  test  forms  when  each  was  used  as  pre¬ 
test  and  as  posttest-1,  and  for  the  two  combined: 


TEST 

MEAN  SCORES 

MEAN  SCORES 

OVERALL 

AS  PRETEST 

AS  POSTTEST-1 

MEAN  SCORE 

Form  A 

6.73  (N-26) 

17.00  (N*27) 

11.96 

Form  B 

9.70  (N«27) 

12.31  (N*26) 

10.98 

The  mean  totals  for  the  two  forms  are  not  significantly  differ¬ 
ent. 


In  a  number  of  the  analyses  we  wish  to  discuss  later,  it  will  be 
necessary  to  make  allowance  for  the  fact  that  all  subjects  did 
not  finish  studying  the  book.  The  principal  reason  for  this  is 
that  the  assignment  was  too  heavy  for  them  to  cover  in  a 
two-week  period  in  addition  to  their  other  work.  For  the 
ninety  per  cent  who  were  employed  during  the  day,  the  topic 
of  probability  must  not  have  seemed  appealing  leisure-time  fare. 
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RESULTS,  continued 


(continued) 

Incomplete 

Subjects 


The  amount  each  subject  did  complete  was  determined  jointly  by 
the  time  log  (where  the  page  coverage  was  recorded)  and  by  the 
evidence  of  completed  feedback  pages. 

The  graph  below  shows  the  percentage  of  subjects  who  completed 
the  various  topics  of  the  book.  Beneath  the  graph,  the  topics 
themselves  are  given  in  order  along  with  the  number  of  pages 
involved  and  the  cumulative  page  total  from  beginning  to  end  of 
the  book. 
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RESULTS,  continued 


(continued) 

Incomplete 

Subjects 


Experts ' 
Scores 


Subjects' 
Scores  Com 
pared  to 
Experts ' 


These  data  will  enter  into  an  analysis  we  shall  describe 
presently  concerning  success  rates  for  the  various  topics. 
Since  we  want  to  compare  our  subjects'  performance  on  these 
topics  with  that  of  the  experts,  we  must  pause  first  now  to 
describe  the  experts'  scores. 


Each  of  the  six  experts  who  served  as  our  comparison  group 
took  the  two  achievement  tests.  Form  A  and  Form  B;  thus  we 
have  twelve  scores  to  average  for  our  reference  point. 

The  highest  possible  score  on  each  test  was  46.  The  mean  of 
the  experts  was  40.16.  The  scores  ranged  from  32  to  46;  the 
median  was  40.5. 


In  the  two  experimental  groups,  Open  and  Closed,  19  subjects 
completed  studying  the  entire  book.  Four  of  these  obtained 
posttest-2  scores  equal  to  the  mean  of  the  expert  group.  Four 
more  scored  between  70%  and  90%  of  the  experts'  mean  score. 
Thus,  8  out  of  the  19  subjects  obtained  scores  exceeding  70% 
of  the  experts'  mean  score.  The  table  below  gives  all  the 
results: 


Percentage  of  Experts'  Mean  Score 

100-91 

90-81 

80-71 

70-61 

Below  61 

Number  of  subjects  scoring 

4 

2 

2 

5 

6 

Per  cent  of  subjects  scoring 

21.1 

10.5 

10.5 

26.3 

31.6 

The  unfinished  subjects  naturally  did  not  do  as  well.  Only  6 
of  the  16  exceeded  50%  of  the  experts'  mean  score. 
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RESULTS,  continued 


Success  Rate 
on  Separate 
Topics 


The  learning  materials  can  easily  be  subdivided  into  separate 
topics  and  a  success  rate  computed  to  see  if  the  sections  are 
equally  effective. 

Pretest  and  posttest-2  scores  were  plotted  for  each  topic 
using  only  the  data  of  those  subjects  who  had  comleted  study 
of  the  given  topic.  Posttest-2  scores  were  used  because  we 
are  more  interested  in  over-all  amount  of  learning  apart  from 
any  skill  in  rapid  retrieval.  For  this  analysis  of  topic 
effectiveness,  Open  and  Closed  groups  were  combined  since 
their  scores  were  very  similar. 

The  graph  below  plots  the  results  in  terms  of  the  mean  percen¬ 
tage  of  subjects  passing  the  items  for  a  given  topic. 


Posttest- 2 


Pretest 


Events  '  Conditional  1  Independence  * 

Measure  Joint  Addition 
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RESULTS,  continued 


(continued) 
Success  Rate 
on  Separate 
Topics 


The  decline  in  scores  from  left  to  right  is  interpreted  as 
indicative  of  the  difficulty  of  the  topic;  for  example,  con¬ 
ditional  probability  is  more  complex  to  grasp  than  is  simple 
event  terminology.  The  fact  that  the  distance  between  pre¬ 
test  and  posttest-2  lines  is  almost  equal  right  across  the 
graph  is  encouraging  evidence  that  the  topic  treatment  has 
been  uniform. 


The  top  line  in  the  graph  plots  the  standing  of  our  group  of 
experts  on  these  various  topics.  For  the  most  part  they 
follow  a  trend  much  like  that  of  our  subjects. 
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RESULTS,  continued 


Time  Data  We  next  took  up  the  question  of  whether  students  who  are  pre¬ 

paring  for  an  open-book  examination  study  as  long  as  do  students 
anticipating  a  closed-book  test. 

The  time  data  were  obtained  from  the  students'  logs. 

Data  bearing  on  this  question  are  of  interest  to  prospective 
users  who  may  wonder  what  time  investments  they  must  make  for 
given  learning  outcomes.  The  information  is  also  of  interest 
as  an  indication  of  students'  reactions  to  the  two  different 
test  prospects . 

In  presenting  the  time  data  for  the  two  subject  groups,  we 
separate  those  who  finished  the  book  and  those  who  did  not: 


SUBJECTS  WHO  FINISHED  BOOK 

Condition 

Mean  Pre¬ 
requisite 
Score 

Mean  Pre¬ 
test 

Score 

Mean  Post¬ 
test-1 
Score 

Mean  Post¬ 
test-2 
Score 

Mean 

Hours 

of 

Study 

Open  (N  =  8) 
Closed  (N-ll) 

15.63 

13  27 

9.27 

9.73 

19.88 

21.73 

28.00 

26.45 

8.08 

10.27 

SUBJECTS 

WHO  DID  NOT  FINISH  BOOK 

Condition 

Mean  Pre- 

Mean  Pre- 

Mean  Post- 

Mean  Post- 

Mean 

requisite 

test 

test-1 

test- 2 

Hours 

Score 

Score 

Score 

Score 

of 

Study 

Open  (N  =  9} 

11.67 

5.22 

14.76 

Esai 

7.43 

Closed  (N=7) 

7.58 

11.44 

EIEB 

8.28 

The  more  informative  and  cleaner  comparisoi  is  to  be  found  in  the 
data  of  the  complete  subjects.  Here  we  see  that  those  in  the 
Closed  group  spent  over  two  hours  longer  in  study  than  did  the 
Open  group  students.  It  is  interesting  to  note  that  in  posttest- 
2  they  scored  almost  as  high  as  did  the  Open  group  who  had  their 
books  to  consult. 

In  this  connection  the  students  made  some  spontaneous  comments 
about  their  attitudes  toward  preparing  for  tests:  several 
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RESULTS,  continued 


(continued)  renarked  that  one  must  study  just  as  much  for  an  open-book 

Tine  Data  test  because  otherwise  it  is  easy  to  become  confused  and  panicky 

leafing  through  a  book  undeT  test  pressures.  At  the  post¬ 
test  sessions,  a  considerable  number  (62%)  of  the  Open  group 
subjects  even  reported  forgetting  that  they  were  tc  have  an 
open-book  test.  Yet  the  group  as  a  whole  did  put  in  less 
study  tine  than  did  the  Closed  group . 

The  data  for  the  unfinished  subjects  have  several  points  of  in¬ 
terest  also.  In  the  Open  group  the  unfinished  subjects 
studied  almost  as  long  as  did  those  who  finished,  so  apparently 
the  fact  that  these  subjects  did  not  get  finished  was  not  from 
neglect  of  the  task.  If  we  look  at  their  other  data,  we  find 
that  these  Open  incomplete  subjects  had  low  scores  on  their 
prerequisite  test  and  on  their  pretest  as  well. 

A  rather  similar  picture  emerges  from  the  data  of  the  incom¬ 
plete  Closed  subjects  --  their  prerequisite  test  scores  were 
not  so  bad  but  they  too  did  poorly  on  the  pretest  for  the 
probability  unit.  They  studied  two  hours  less  than  did  the 
members  of  their  group  who  finished  the  task.  In  a  later  sec¬ 
tion  we  shall  look  into  the  relation  of  test  scores  and  the 
mathematical  background  of  the  students. 

Briefly,  then,  the  time  data  show: 

•  The  probability  and  set  units  require  about  8  hours  of 
study  for  an  open-book  test  and  10  hours  of  study  for 
a  closed-book  test. 

•  Students  who  are  not  so  well  prepared  mathematically  can 
be  expected  to  take  longer. 

Open  Book  We  mentioned  above  that  some  students  remarked  that  they  did 

Use  During  Test  not  rely  on  the  book  much  during  open-book  tests  and  that  some 
of  our  students  even  forgot  that  they  were  to  get  an  open- 
book  test. 

Fortunately  after  the  test  we  gave  the  Open  group  an  extra 
short  questionnaire  which  asked  in  what  per  cent  of  the  test  ques¬ 
tions  did  the  student  use  the  book. 

The  answers  ranged  from  zero  to  35%.  The  10%  mark  divides 
the  subjects  into  approximately  equal  groups.  In  the  table 
below  the  scores  obtained  by  those  who  used  the  book  more  can 
be  compared  with  those  of  subjects  who  used  it  less  (as  usual, 
finished  and  unfinished  subjects  are  separated) : 
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RESULTS  ,  continued 


(continued) 

Open  Book 
Use  During  Test 


r . 

Open  Group  Subjects  Who  Had  Finished  Book 

Use  of  Book 

N 

Mean 

Prerequisite 

Score 

Mean 

Pretest 

Score 

Mean 

Posttest-2 

Score 

Posttest-2 
Expressed 
as  Percentage 
of  Experts 

Over  10% 

a 

16.0 

9.22 

32.0 

10%  or  less 

B 

15.25 

10.00 

24. C 

Open  Group  Subjects  Who  Had  Not  Finished  Book 

Use  of  Book 

N 

Mean 

Prerequisite 

Score 

Mean 

Pretest 

Score 

Mean 

Posttest-2 

Score 

Posttest-2 
Expressed 
as  Percentage 
of  Experts 

Over  10% 

| 

12.4 

4.50 

19.4 

10%  or  less 

10.8 

0.25 

.  ...... 

13.5 

For  both  finished  and  incomplete  subjects,  those  who  used  the 
book  more  during  the  test  were  more  successful. 


Sid>jects'  Just  before  the  posttest,  subjects  wrote  down  the  grade  they 

Goals  expected  to  obtain  on  the  test.  If  we  take  this  as  some 

indication  of  the  goals  the  subjects  set  for  themselves,  it 
is  interesting  to  see  how  well  they  succeeded. 

In  the  table  below  we  have  grouped  the  subjects'  grade  esti¬ 
mates  into  three  classes:  the  first  represents  A's  and  B's 
(only  one  subject  expected  an  A);  the  second,  C's;  and  the 
third,  D's  and  below.  The  body  of  the  table  shows  the  mean 
posttest-2  scores  expressed  as  percentages  of  the  experts' 
mean  score. 


Scores 

Predicted 

By  Subjects 

Open  Group 

Closed  Group 

Total 

N 

% 

N 

% 

N 

% 

81  -100% 

5 

78.0 

6 

71.3 

11 

74.3 

71-80% 

8 

48.5 

6 

44.6 

14 

46.8 

Below  71% 

3 

42.5 

4 

41.0 

7 

41.6 
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RESULTS  •  continued 


(continued) 

Subjects' 

Goals 


Those  who  expected  to  score  between  80  and  100  on  the  material 
emerged  with  a  fairly  high  score,  74.3%  -  which  falls  short 
of  their  aspirations,  yet  is  a  respectable  showing  on  a  diffi¬ 
cult  test  after  only  two  weeks'  exposure  to  the  topic. 

Those  who  expected  to  make  a  C  on  the  test  were  considerably 
short  of  the  mark.  Only  six  of  the  fourteen  in  this  group  had 
finished  the  book,  so  one  wonders  at  the  confidence  of  those 
who  expected  to  score  70-80  without  having  finished  it. 

Even  those  six  who  did  finish  it  did  not  do  as  well  as  they 
hoped  -  their  average  was  52.5%  of  the  experts. 

Those  who  anticipated  doing  poorly  on  the  test  were  quite  right; 
their  score  was  only  41.6%  of  the  experts.  Five  of  the  seven 
in  this  group  had  not  finished  the  book. 

The  pre-study  attitude  survey  asked  subjects  to  indicate  the 
grade  they  usually  aspired  to  in  their  mathematics  courses. 

If  we  separate  the  subjects  into  three  approximately  equal 
groups  on  the  basis  of  their  responses  and  then  look  at  'the 
posttest-2  scores  for  each  of  these  groups,  we  get  the 
following  table: 


Usual  Grade 

N 

Posttest-2  Means  as  Per- 

Aspirations 

centage  of  Experts'  Mean 

91-100 

9 

70% 

81  -90 

12 

51% 

Below  81 

8 

36% 

Naturally  if  we  single  out  those  who  both  had  high  expectations 
on  this  particular  test  and  generally  expect  to  do  well  in 
mathematics  courses,  we  find  a  high-scoring  group  -  average 
78%  (N  »  4)  on  posttest-2  scores.  Those  who  did  not  expect  to 
do  well  either  here  o£  in  their  courses  came  off  very  poorly  - 
29%  (N  *  3).  It  is  not  feasible  to  consult  other  interesting 
combinations  of  the  t*'o  goals  indicators  because  the  number 
of  cases  becomes  even  smaller. 

In  general  the  results  on  the  probability  test  do  follow  with 
measures  of  the  streets'  expectations  (at  least  in  terms  of 
the  ranking  of  the  means).  Among  the  important  determinants 
of  the  goal  indicators  will  be  the  students'  general  scho¬ 
lastic  aptitude  and  his  level  of  competence  in  mathematics 
at  the  time  he  came  to  the  project. 
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RESULTS,  continued 


(continued) 

Subjects* 

'  Goals 


Mathematics 

Background 


The  latter  would  be  an  especially  strong  influence  presumably 
in  the  grade  expected  for  the  probability  test.  We  examine 
the  students*  scores  in  relation  to  theiT  mathematics  back¬ 
ground  in  the  next  section. 


The  prerequisite  test  gave  us  a  measure  of  our  subjects'  present 
proficiency  in  simple  math  and  algebra  problems  while  their 
personal  data  record  provided  a  listing  of  the  college  math 
courses  they  had  taken. 

From  these  two  kinds  of  information  we  derived  a  loose  measure 
of  mathematics  background  by  this  procedure: 

.  for  each  kind  of  information  a  cutoff  point  was  arbitrarily 
'chosen  to  separate  the  subjects  into  a  high  and  a  low  group. 
For  instance,  for  the  prerequisite  test,  those  scoring  over 
75%  were  called  "high,"  those  under  75%  went  into  the  "low" 
group.  In  terms  of  mathematics  courses,  those  who  had 
taken  calculus  and  beyond  were  called  "high,"  those  who  had 
not  had  college  calculus  were  put  in  the  "low"  group. 

.  for  a  composite  mathematics  background  measure,  we  placed 
in  a  High  group  those  who  were  high  on  both  of  the 

above  dimensions.  Those  low  on  both  went  into  a  Low 

group,  while  those  who  were  high  on  one  but  low  on  the  other 
were  assigned  to  a  Medium  group. 

This  classification  resulted  in  twelve  subjects  in  .the  High 
group,  eight  in  the  Medium  group,  and  fifteen  in  the  Low  group. 
Since  different  members  in  each  group  finished  the  book,  the 
results  in  this  section  will  be  based  on  complete  subjects 
only.  This  reduces  the  number  of  subjects  in  the  High  group 
to  nine.  Medium  group  to  three,  and  Low  group  to  eight.  Only 

25%  of  the  High  group  failed  to  complete  the  book  while  almost 

50%  of  the  Low  group  did  not  finish.  This  seems  to  be  a  result 
of  the  two  week  time  limit  imposed  since  the  High  group 
(N  =  12)  spent  an  average  of  7.9  hours  on  the  Dook  while  the 
Low  group  (N  =  15)  spent  an  average  of  9.5  hours. 

The  probability-unit  results  for  each  group  are  shown  next 
for  those  subjects  who  finished  the  book  (Open  and  Closed 
groups  pooled) .  The  same  data  are  shown  in  the  accompanying 
graph. 
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RESULTS,  continued 

(continued)  Those  who  are  better  prepared  in  mathematics  do  score  higher 
Mathematics  on  the  pretest  and  posttests  than  do  those  in  the  Low  group. 

Background  It  is  interesting  to  note,  however,  that  the  gain  scores  for 

the  groups  are  not  dissimilar: 


Aptitude 

Group 

■MMHMKaMMHMMMUaMKl 

High  Math  (N=9) 
Medium  Math  (N=3) 
Low  Math  (Nr 8) 


Mean  Gain  from  Pre-  Mean  Gain  from  Pre¬ 
test  to  Posttest-1  test  to  Posttest-2 


Reference  Use 
Of  Information 
Maps 


On  a  logical  basis  the  advantages  of  information  map  books  for 
reference  purposes  seem  obvious.  Experimental  evidence  on  the 
question,  however,  is  not  easy  to  come  by  because  it  is  diff¬ 
icult  to  design  realistic  reference  situations  that  are  not 
"straw-man"  demonstrations  of  information  map  superiority.  We 
do  have  a  few  data  that  bear  on  the  question,  however. 

The  Harvard  series  had  shown  us  that  students  in  initial  learn¬ 
ing  of  a  short  course  where  the  grade  is  inconsequential  do  not 
leaf  back  and  forth,  reviewing,  comparing,  integrating  as 
they  do  in  a  semester- length  college  course.  Thus  their  exper¬ 
ience  is  not  a  very  strong  base  from  which  to  judge  the  book’s 
utility  as  a  reference  tool  Consequently  we  decided  to 
incorporate  in  the  next  tryout  a  task  that  would  require 
moving  back  and  forth  through  the  book. 

The  Open-book  test  for  the  probability  course  was  made  part  of 
the  experimental  plan  in  order  to  obtain  data  on  the  book's  use 
in  reference  for  a  real  task  -  answering  test  questions. 

We  have  already  reported  that  a  disappointingly  small  number 
of  students  actually  used  the  book  during  the  test.  And  we 
have  also  reported  that  those  who  did  use  the  book  scored 
higher  than  those  who  used  it  very  little. 

For  these  same  subjects  we  have  responses  to  a  short  posttest 
questionnaire  about  their  experience  in  looking  up  things  in 
the  book.  We  asked: 

"When  compared  with  standard  textbooks,  looking  up 
things  in  the  information  map  book  is  . 


Of  the  seventeen  subjects  in  the  Open  group,  one  wrote  "no 
opinion,"  two  felt  the  books  were  the  same,  and  fourteen  wrote 
that  the  map  book  was  "easier"  or  "better"  or  some  such  phrase, 
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RESULTS,  continued 


(continued) 
Reference  Use 
Of  Information 
Maps 


Many  of  the  latter  group  added  consents  such  as: 

"interesting  --  easy  to  find  —  usually  exactly  what 
one  is  looking  for." 

"much  easier  and  very  direct." 

"very  easy  reference." 

"it  was  clear  and  easier  to  use  because  I  could  follow 
through  with  examples." 

"easier.  This  is  more  concise  and  with  the  Related 
Pages  much  easier." 

"much  easier  and  quicker.  I  can  go  right  to  the  page 
to  find  the  needed  information." 

Those  who  had  used  the  book  more  during  the  test  and  those  who 
rarely  used  it  were  equally  favorable  in  answering  this  question. 
TVo  of  the  three  subjects  who  were  noncomital  or  neutral  on 
the  comparison  were  from  the  High  mathematics  background  group, 
and  the  third  was  from  the  Medium  group.  But  the  fourteen  who 
responded  favorably  were  almost  equally  distributed  over  the 
High,  Medium,  and  Low  groups  (4,5,5,  respectively).  Thus 
opinion  about  the  book's  utility  for  reference  was  not  related 
to  the  mathematics  background  of  the  users. 


RESULTS,  continued 


Attitude 

Results 


Since  the  attitude  questionnaire  for  this  stucfy  was 
longer  than  the  ones  used  in  the  two  previous  studies,  it  is 
easier  to  present  the  results  in  two  separate  parts:  attitude 
towards  learning  features  of  information  mapping  and  attitude 
towards  format. 

The  questions  about  learning  features  give  the  following  results 

1.,  89%  thougit  that  learning  sets  with  the  materials  was 
effective;  one  (out  of  the  35  subjects)  thought  that  the 
material  was  ineffective;  the  rest  were  undecided. 

2.  63%  though^  that  learning  probability  with  the  material 
was  effective;  9%  thought  the  material  was  ineffective, 
and  the  rest  were  undecided. 

3.  75%  thought  they  would  retain  the  material  presented  better 
than  with  a  standard  text,  two  subjects  disagreed  with  this, 
and  the  rest  were  undecided. 

4.  85%  would  recommend  the  materials  to  others,  one  subject 
would  not,  and  the  rest  were  undecided. 

5.  When  asked  whether  the  material  progressed  too  quickly, 

26%  agreed,  29%  disagreed,  and  the  remaining  45%  said  it 
was  "just  the  right  pace." 

6.  The  section  on  sets  was  rated  easy  by  89%  of  the  subjects, 
and  just  right  by  the  remaining  11%. 

7.  The  probability  section  was  rated  easy  by  17%  of  the  sub¬ 
jects,  just  right  by  20%,  and  difficult  by  the  other  63%. 
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RESULTS,  continued 


(continued)  The  questions  relating  to  the  information  map  format  evoked 
Attitude  these  responses : 

Results 

S.  37%  of  the  subjects  agTeed  that  there  were  not  enough  feed¬ 
back  questions  and  51%  disagreed;  the  rest  were  undecided. 

9.  89%  of  the  subjects  thought  the  feedback  questions  were 

effective,  one  thought  they  weren’t,  and  three  were 
undecided. 

10.  80%  were  interested  in  studying  other  subject  matter  written 
in  this  style,  one  subject  didn’t  care  one  way  or  the  other, 
and  the  others  were  not  too  interested. 

11.  61%  felt  that  they  were  able  to  work  with  the  book  for 
longer  periods  of  time  than  with  a  standard  textbook.  14% 
disagreed  with  this  statement  and  the  other  25%  thought 
there  was  no  difference. 

12.  86%  thought  that  the  use  of  diagrams  simplified  the  material, 
two  of  the  subjects  disagreed  with  the  statement,  and  the 
other  three  were  neutral. 

13.  69%  thought  the  examples  in  the  book  were  interesting,  9% 
thought  them  uninteresting,  and  the  other  22%  had  no 
opinion. 

In  addition  to  these  multiple  choice  questions,  there  were  a 
number  of  open-ended  questions  which  asked  about  various  fea¬ 
tures  of  the  material. 

When  asked  to  list  the  ways  this  book  was  different  from  a  stan¬ 
dard  text,  some  of  the  responses  and  the  nunber  of  subjects  who 
gave  each  were: 

talked,  asked  questions,  and  explained  answers  (7) 

*  clear  examples  (2) 

*  more  simplified  and  clarified  (5) 

'  logical  grouping  of  ideas  (16) 

*  repetitious  (2) 

*  immediacy  of  feedback  questions  (11) 

*  easier  to  stop  and  start  studying  (1) 

*  explained  material  better  (1) 

We  also  asked  which  features  of  the  book  were  not  helpful: 

*  some  introductions  too  long  (2) 

*  explanations  too  involved  (1) 

*  confusing  formulas  (1) 

two  week  time  limit  too  short  (1) 

*  having  to  work  completely  independently  (1) 
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RESULTS,  continued 


(continued) 

Attitude 

Results 


General  Dis¬ 
cussion 


Some  of  the  responses  to  what  features  of  the  book  were 
helpful : 

.  diagrams  explaining  formulas  (4) 

.  numerous  examples  (5) 

.  alternative  approaches  to  a  given  problem  (3) 

.  feedback  questions  (13) 

.  reviews  with  answers  given  (11) 

.  division  into  concepts  (5) 

When  asked  how  we  could  improve  the  book  we  got  the  following 
responses : 

.  add  more  interesting  examples  (2) 

.  add  more  review  sections  (2) 

.  add  more  detailed  explanations  (2) 

.  add  more  examples  on  complex  material  (7) 

.  add  more  time  on  complex  probabilities  (4) 

.  add  more  feedback  questions  (2) 

.  delete  some  of  the  material  on  sets  (2) 

Some  of  the  individual  comments  were  (the  letters  H,  M,  and 
L  refer  to  the  subject's  math  group  -  high,  medium  or  low): 

•  first  math  "course"  that  ever  got  through  my  mental 
block  (L) 

•  with  this  text  I  was  secure  with  a  math  book.  This 
is  phenomenal  for  me!  (L) 

•  much  more  boring  and  repetitious  than  a  standard  text  (H) 

•  more  fun  to  read  (M) 

•  reader  more  active  in  learning  (M) 

•  compared  to  other  math  books  I  found  this  to  be  a  great 
improvement  (L) 

•  extremely  repetitious  (H) 

•  bored  with  approaching  each  topic  in  the  same  way  - 
organized  but  not  stimulating  (L) 

•  missed  having  a  teacher  for  aid  (L) 

•  don’t  like  feeling  like  ao.  automation  machine  -  want  to 
select  information  not  react  to  it  (H) 

•  easy  to  work  with;  efficient  means  of  learning  (H) 


Based  on  their  answers  to  all  the  attitude  questions,  thirty 
of  the  thirty-five  subjects  (85.77%)  had  a  favorable  reaction 
to  the  learning  materials,  three  were  indifferent,  and  two  had 
an  unfavorable  reaction.  The  subjects  who  were  weak  in  math¬ 
ematics  were  very  favorably  disposed  and  perhaps  would  have 
done  better  on  the  achievement  test  if  they  had  more  time  to 
study. 
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RESULTS,  continued 


Most  of  the  unfavorable  comments  came  from  those  subjects  who 
were  quite  good  in  math.  Their  usual  comment  was  that  expla¬ 
nations  were  too  long,  boring,  etc.  When  we  looked  at  their 
books  it  was  evident  that  they  had  gone  through  all  the  ma¬ 
terial  and  all  the  feedback  questions  even  though  we  stated 
in  the  introduction  that  the  book  was  designed  so  that  examples 
could  be  skipped  and  feedback  questions  answered  only  when  the 
student  was  having  difficulty. 

The  few  unfavorable  comments  of  those  who  had  weak  math  backgrounds 
were  generally  confined  to  lack  of  time  and  the  desire  to  ha* 
a  teacher  available  to  answer  questions  that  came  up  so  the 
could  go  on  with  confidence. 

Limitations  of  The  principal  shortcoming  of  this  study  lies  in  the  atypical 
the  Study  motivational  conditions .  One  of  the  key  principles  of  modem 

learning  research  relates  learning  and  memory  to  the  degree  of 
motivation  aroused.  Among  the  important  incentives  commonly 
employed  to  motivate  students  are  grades  and  the  approval  of 
the  instructor.  In  the  laboratory  situation,  monetary  rewards 
are  often  used  to  manipulate  the  subjects'  motivational  level. 

In  our  evaluative  experiment  with  the  probability  book,  these 
common  incentives  were  absent.  The  grades  the  student  achieved 
on  our  tests,  were  of  no  consequence  to  him,  would  not  affect 
the  future  course  of  his  educational  career  one  bit  and 
in  fact  would* never  even  become  known  to  him.  Although  he  was 
paid  for  coming  to  take  part  in  the  series,  his  rate  of  pay  was 
in  no  way  geared  to  his  level  of  achievement;  as  we  saw,  many 
subjects  did  not  finish  reading  through  the  materials.  There 
was  not  the  "cramming"  for  an  exam  that  so  often  typifies 
(unfortunately)  the  college  course. 

The  time  of  year  when  this  series  occurred  was  one  where  stu¬ 
dent  morale  and  enthusiasm  for  study  were  at  their  lowest  ebb  -- 
at  the  end  of  the  school  year  just  after  final  exams.  Most 
subjects  had  just  started  on  their  summer  jobs. 

What  motives  were  operating  to  make  them  take  part  in  the  ex¬ 
periment  and  to  study  the  book  as  many  hours  as  they  did 
(5  to  14)?  We  know  that  many  came  because  the  project  seemed 
a  painless  and  interesting  way  to  earn  some  much  needed  money; 
some  came  as  a  favor  to  the  investigators;  a  couple  came  be¬ 
cause  they  saw  an  opportunity  to  prepare  for  a  similar  course 
required  next  term  by  their  college  program. 
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General 

Discussion 


RESULTS,  continued 


(continued) 
Limitations 
of  the  Study 


Whatever  may  be  the  needs  and  interests  that  drew  our  subjects 
to  participate  in  the  study,  we  are  confident  that  they  were 
less  intense  than  the  motives  ordinarily  operating  in  the 
situations  for  which  these  learning  materials  were  intended. 

Of  our  results  it  is  safe  to  say  that  they  represent  outcomes 
obtained  under  minimal  motivational  conditions.  In  other  cir¬ 
cumstances*  (in  college  courses,  job-training  classes,  etc.) 
where  internal  and  external  incentives  are  operating  at  cus¬ 
tomary  levels  and  where  sufficient  time  is  available  for 
assimilation,  we  can  reasonably  predict  markedly  increased 
effects . 
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NOTE  ON  STATISTICAL  DETAILS 


The  Multiple 
Analysis  of 
Covariance 


The  multiple  analysis  of  covariance  used  the  prerequisite 
test  QL)  and  the  pretest  (X~)  as  covariates  for  the  posttest 
(Y) .  Tne  analysis  of  sums  or  squares  and  products  was  as 
follows : 


d.f . 

EXj2 

ex22 

Ex1x2 

Between  groups 
Within  groups 
Total 

2 

50 

52 

1.63 

1141.12 

1147.75" 

21.71 

2700.10 

mnsi 

-5.95 

897.27 

891.32 

d.f. 

Ey2 

Exjy 

z*2y 

Between  groups 

2 

787.09 

6.89 

-21.79 

Within  groups 

50 

3512.08 

1152.63 

2166.72 

Total 

52 

4299.17 

1159.52 

2144.91 

The  regression  coefficients  are  estimated  by  solving  the 
simultaneous  equations: 

1141.12b,  ♦  897.27b.  *>  1152.63 
897. 27b l  ♦  2700.1052  *  2166.72 

which  results  in: 


bi  *  .513 
b2  =  .632 

The  deviations  sums  of  squares  weTe: 


d.f. 

SS 

MS 

Total 

50 

2348.68 

Error 

48 

1551.36 

32.32 

For  testing  adjusted  means 

2 

797.32 

L.— -  - 

398.66 

The  F  test  of  the  adjusted  means  results  in: 
F  =  12.33  (2,48  d.t.;  P  <  .001) 
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NOTE  ON  STATISTICAL  DETAILS,  continued 


(continued) 
The  Multiple 
Analysis  of 
Covariance 


The  effective  mean  square  per  observation  is: 
s2  =  32.72 

(This  is  an  adjusted  error  mean  square  -  see  Snedecor  and 
Cochran,  1967,  p.  441) 

The  Studentized  Range  Test  (Q)  may  be  use$l  to  compare  the 
differences  between  adjusted  means.  The  upper  1%  point  of 
Q  for  three  means  and  48  d.f.  is  approximately  4.33.  The 
average  standard  error  of  a  mean  is  1.39.  Thus,  any  differ¬ 
ence  which  is  greater  than  (4. 33) (1.39)  *  6.019  is  significant 
at  the  1%  level.  Since  the  difference  between  open  and 
control  is  8.38  and  the  difference  between  closed  and  control 
is  8.17,  both  of  these  differences  are  significant  at  the 
1%  level. 
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CHAPTER  7  OVERVIEW  OF  THE  EVALUATIVE  STUDIES 


Introduction 


Achievement 

Data 


First  of  all,  two  points  about  evaluative  studies  in  general: 

•  The  experimental  data  bear  on  the  specific  product 
and  not  on  the  method.  Just  as  one  given  textbook 
cannot  be  considered  as  representative  of  so  broad 
a  category  as  "textbooks  in  general,"  so  the  in¬ 
formation  map  units  we  have  tested  are  only  one 
sample  of  the  possible  products  of  the  information 
map  method. 

There  are  limits  to  what  can  be  accomplished  by  one 
series  of  researches  on  any  topic. 


The  evaluative  studies  have  shown  some  of  the  effects  that 
we  can  expect  from  the  use  of  this  information  map  book  under 
different  conditions.  After  8  to  10  hours  of  study  in  a 
setting  free  of  the  usual  educational  pressures,  students 
showed  significant  evidence  of  learning.  That  fact  in  itself 
is  not  surprising  in  view  of  the  empirical  tryouts  and  re¬ 
visions  that  the  program  underwent  throughout  its  development. 

The  problem  of  trying  to  say  anything  meaningful  about  the 
magnitude  of  gains  from  learning  materials  is  a  perennial 
one  in  educational  research.  The  arbitrariness  both  of  the 
criteria  set  by  the  program  developers  and  of  the  difficulty 
level  of  the  tests  make  it  impossible  to  compare  gains  across 
evaluative  studies.  One  set  of  criterion  tests  may  be  very 
easy  and  a  spectacularly  high  percentage  of  students  pass  them 
compared  with  the  gains  from  another  program  with  more  diffi¬ 
cult  criterion  tests. 

In  short,  since  the  criteria  and  the  difficulty  level  of  the 
tests  are  the  result  of  the  individual  developer’s  judgment, 
much  of  the  data  from  evaluative  studies  are  properly  termed 
"ipsative"  -  self-serving,  without  an  external  reference  point. 

Our  own  test  of  the  probability  unit  were  purposefully  made 
difficult  for  reasons  of  experimental  design,  as  Chapter  6 
explains.  In  the  absence  of  any  standard  achievement  tests 
to  use  for  comparison,  it  is  difficult  to  assess  the  size  of 
the  gains. 

In  an  effort  to  provide  some  external  anchor  point  for  judging 
the  effect  of  the  probability  unit,  we  resorted  to  comparing 
the  students'  scores  wit’'  those  of  our  "experts"  on  the  same 
test. 


continued  on  next  page 
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(continued) 

Achievement 

Data 


Attitude 

Data 


Conclusion 


* 


To  further  aid  in  weighing  the  value  of  the  program,  we 
present  in  the  appendix  the  actual  tests  used  for  the  prob¬ 
ability  unit.  There  the  reader  may  note  the  range  of  content 
covered  and  the  difficulty  level  of  the  material.  These  to¬ 
gether  with  the  performance  data  reported  in  Chapter  6  consti¬ 
tute  the  main  evidence  for  the  unit's  value. 

In  the  end,  whether  or  not  the  program  has  achieved  worthwhile 
results  is  a  matter  of  individual  subjective  judgment,  and 
that  is  related  to  the  use  for  which  the  specific  product 
is  being  considered. 


The  difficulty  of  trying  to  establish  an  objective  standard 
for  comparing  educational  programs  has  led  some  researchers 
to  turn  to  attitude  data  as  being  more  revealing  about  the 
attractive  power  of  a  program.  Although  we  do  not  abandon 
hope  of  solving  the  methodological  issue,  we  do  attach 
importance  to  the  user's  reactions  to  the  learning  materials. 
His  willingness  to  approach  and  to  interact  with  the  materials 
encourages  the  hope  that  the  materials  may  communicate  with 
him. 

The  reactions  of  our  students  to  information  maps  were  gen¬ 
erally  quite  favorable,  especially  among  those  for  whom  the 
materials  were  intended,  those  with  weak  backgrounds  in 
mathematics. 

In  general,  our  subjects  rated  the  materials  effective,  said 
they  would  recommend  them  to  others,  and  singled  out  many  of 
the  special  features  for  favorable  mention. 

Whether  the  effects  of  the  sets  and  probability  book  are 
attributable  to  the  novelty  of  the  method  cannot  be  determined 
in  the  time -framework  within  which  we  operated.  The  so-called 
Hawthorne -type  reactions  can  be  both  hostile  and  enthusiastic. 
Only  in  extended  tryouts  with  the  same  subjects  can  we 
determine  whether  the  learning  materials  are  still  looked  upon 
favorably  by  students. 


In  working  with  a  new  system,  we  have  made  only  a  beginning 
at  collecting  information  about  how  subjects  react  to  the 
learning  materials.  We  have  sampled  only  a  narrow  range  of 
subject  matters.  We  have  only  begun  to  ask  questions  about 
how  specific  factors  or  features  of  the  materials  contribute 
to  the  learning  outcomes.  One  set  of  experiments  with  a  single 
product  can  only  begin  to  map  out  the  areas  that  may  be 
important. 


continued  on  next  page 
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(continued) 

Conclusion 


Our  initial  task  was  to  explore  the  effects  of  information- 
napped  learning  materials  and  to  decide  upon  the  feasibility 
of  further  work  with  the  method.  The  initial  research  is 
promising  but  more  experiments  and  wider  experience  are  needed 
if  we  are  to  be  able  to  generalize  about  the  method.  This  is 
not  to  say  that  a  strong  logical  case  cannot  be  made  for  the 
method's  advantages.  We  will  consider  this  in  the  final  chapter. 
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CHAPTER 


INFORMATION  MAPPING  AND  COMPUTERS 


A  LEARNING-REFERENCE  SUBSYSTEM 


Introduction  During  the  development  of  information  map  books,  we  began  to 
see  how  the  information  mapping  categories  could  be  applied  to 
organizing  a  computer  data  base  for  learning  and  reference 
purposes.  We  explored  some  of  the  aspects  of  developing  a 
multi-purpose  data  base  organized  in  this  way  and  we  carried 
out  exploratory  design  work  on  animated  display  possibilities. 
Some  preliminary  tryouts  of  learning  from  information  map 
displays  re  made  with  a  simulated  display  unit. 

Background  The  capabilities  of  a  large  computer  facility  permit  it  to 
be  a  significant  training  vehicle  for  its  users.  Therefore, 
it  is  desirable  for  such  a  facility  to  include  a  learning- 
reference  subsystem.  The  success  of  such  an  information 
utility  depends  in  a  very  critical  way  on  the  organization  of 
the  underlying  data  base  and  the  flexibility  with  which  it 
can  serve  different  purposes. 

Data  Base  We  conceived  of  a  data  base  as  consisting  of  interrelated 

networks  of  information  segments.  This  gives  a  flexibility 
in  using  only  those  parts  of  the  system  that  are  required  for 
a  particular  purpose.  Since  information  maps  are  composed  of 
separable  labelled  blocks  of  information,  they  can  easily  be 
adapted  for  this  type  of  data  base. 

Sequence  A  common  data  base  to  serve  varied  purposes  would  require  a 

Generators  set  of  "sequence  generators".  These  would  be  rules  or  patterns 

for  assembling  and  displaying  information  blocks,  depending 
upon  the  objective  of  the  user.  The  browser  would  obviously 
not  want  to  see  the  information  in  the  same  sequence  as  would 
the  user  with  a  specific  reference  problem.  The  purposes 
of  initial  learning,  reviewing,  briefing,  updating,  and  so 
on  would  all  require  different  information  sequences.  The 
information  map  books  are  arranged  according  to  sequence 
generators  derived  from  considerations  of  learning  theory, 
instructional  technology,  and  human  factors  engineering. 

For  some  purposes,  the  sequence  generator  could  allow 
considerable  control  of  the  system  by  the  user  —  for  reviewing, 
browsing,  and  reference  uses,  primary  control  should  reside 
with  the  user.  For  initial  learning,  the  sequence  generator 
could  permit  a  limited  number  of  user  options,  but  major  control 
should  probably  remain  with  the  system. 


continued  on  next  page 
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A  LEARNING-REFERENCE  SUBSYSTEM,  continued 


Query  User  control  of  the  system  is  provided  by  the  query  language 

Language  which  is  the  set  of  all  commands  that  are  available  to  the 

user.  For  example,  the  user  may  have  available  commands  which 
allow  him  to  look  up  a  particular  page  or  topic,  go  to  a  set 
of  feedback  questions  (or  omit  such  questions),  etc. 

The  extent  of  user  control  in  the  system  is  indicated  by  the 
complexity  of  the  query  language. 


Updating  From  time  to  time,  there  will  arise  a  need  to  modify  the 

data  base.  An  information  mapped  data  base,  because  of 
its  separate  blocks  of  information,  would  probably  minimize 
the  cost  of  updating  or  improving  the  data  base. 
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ADVANTAGES  OF  USING  INFORMATION  MAPPING  FOR  A  LEARNING-REFERENCE  SUBSYSTEM 


Multiple  The  flexible  block-identified  data  base  can  be  rearranged 

Purposes  for: 

•  initial  learning 

•  for  the  naive  student 

•  for  the  sophisticated  student 

•  relearning  or  review 

•  for  a  comprehension  test 

•  for  a  performance  test 

•  reference  (reminder  or  look-up  of  a  specific  bit  of 
information) 

•  briefing  and  browsing  (what  is  this  subject  matter 
about  in  general  terms?) 

•  updating  skills  and  information  (unlearning  and  new 
learning) 

•  job-aid  (preparing  checklists,  menus,  and  other 
types  of  job  aids) 

User  The  user  may: 

Options 

•  exercise  a  preference  in  sequencing  of  information 
blocks 

•  choose  different  levels  of  feedback  questions 

•  choose  when  to  see  the  feedback  question  answers 

•  choose  whether  he  wants  personal  or  system  control 
of  review 

choose  details  of  a  personal  review  method  (e.g.,  via 
feedback  questions, condensed  summary  pages,  definitions 
only,  etc.) 

System  The  system  can  respond  to  individual  differences  such  as 

Options  learner  purposes.  It  can  give  mathematical  proofs  to  the 

mathematically  inclined  learner  and  omit  them  for  the  person 
who  is  only  interested  in  learning  the  subject  matter 
procedural ly. 


DISPLAY  OPPORTUNITIES 


Introduction  Computers  with  display  scopes  have  great  potential  for  making 
learning  and  reference  work  more  effective.  Considerable 
research  is  needed  to  explore  the  advantages  of  different 
scope  applications. 

The  capacity  of  the  computer  to  permit  the  user  to  exercise 
his  individual  preferences  may  be  an  important  part  of  effec¬ 
tive  man-machine  partnership.  One  option  that  could  be  made 
available  to  the  user  would  be  the  choice  of  a  number  of 
types  of  displays. 

Not  everyone  wants  to  have  information  displayed  to  him  in 
the  same  way.  For  example,  if  you  are  learning  something  for 
the  first  time  you  may  want  a  complete  version  of  the  text. 

If  you  are  reviewing  or  just  browsing  through  a  large  body 
of  material  you  may  want  less. 

Many  people  like  tables,  charts,  and  graphs,  while  others 
prefer  their  information  in  uninterrupted  prose.  Information 
napping  can  easily  be  adapted  to  accommo^  te  both. 


Dynamic  Another  promising  possibility  is  the  development  of  animated 

Displays  displays  along  with  research  into  their  impact  on  learning 

and  reference  work.  Five  possible  uses  of  dynamic  displays 
are  given  below: 


Uses 

Description 

Possible  Advantages 

Guided 

reading 

In  the  guided  reading  type,  we  reveal 
parts  of  a  display  to  the  user.  For 
example,  different  parts  of  a  table 
might  be  shown. 

•  Ensures  that  new  users 
of  information  maps 
use  them  to  full 
advantage. 

•  Paces  reading  rate. 

•  Focuses  attention. 

Vary  level 
of  detail 

In  this  type,  different  levels  of 
detail  or  abstraction  appear  during 
the  display.  For  example,  a  tree  is 
shown  with  just  the  names  of  main 
concepts,  then  with  the  next  sorting 
into  finer  level  groups. 

•  The  user's  field  of 
vision  is  not  over¬ 
clogged  with  detail. 

•  Relationships  between 
different  levels  of 
abstraction  and  detail 
are  revealed. 

continued  on  next  page 
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DISPLAY  OPPORTUNITIES,  continued 


Uses 

Description 

Possible  Advantages 

Show  move¬ 
ment  in  a 
sequence 
network 

In  sequence  networks,  there  is  a 
movement  either  logically  or  in  time. 
We  can  display  this  movement  in 
a  dynamic  display.  We  can  also  show 
interrelationships  between  two  mov¬ 
ing  systems. 

.  Highlight  thing  being 
talked  about. 

.  Focus  on  parts  and 
relationships  in 
whole. 

Explode, 
implode 
parts  of 
structure 

Structures  are  made  up  of  parts. 
Diagrams  can  be  "exploded"  by  a 
common  drafting  technique  in  dynamic 
terms. 

•  Show  how  struc¬ 
tures  fit  together. 

Highlight 

an 

important 

learning 

point 

Flashing  (or  momentarily  appearing) 
arrows,  labels,  and  other  visual 
techniques  adaptable  from  the  world 
of  movie  and  television. 

•  Focus  attention  of 
user. 

•  Maintain  attention  of 
user. 
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ONE  APPLICATION  OF  DYNAMIC  DISPLAYS 


Time 

Segment  Display  Screen 


Comment 


Name  of  table,  main 
headings,  and  outline 
of  table  flash  onto 
screen. 


In  him 

- ..... 

. . -  . 

MUUM  m. 

1 U‘»  MIIU 

Mu 

Names  bn  left  axis  appear 
while  names  of  main 
headings  recede  (but  do 
not  disappear). 


Successive  horizontal 
blocks  of  information 
appear. 

As  each  "line"  of  blocks 
appears,  the  other  lines 
recede  to  a  lesser 
character  size  or  are 
reduced  to  a  lesser 
brightness . 


EXAMPLES  OF  DISPLAY  OPTIONS 


Introduction 


Example 


The 'next  two  sample  pages  illustrate  how  the  same  information 
can  be  presented  in  two  different  ways.  Even  for  initial 
learning  purposes  some  students  prefer  a  chart  where  they  can 
observe  the  relationships  themselves,  while  others  would  '-rather 
have  a  verbal  description  of  the  material.  For  lat^r  review 
and  reference  work,  however,  the  tabular  format  would  pre¬ 
sumably  be  most  favored. 

The  first  example  shows  the  summary  table  from  the  end  of 
the  sets  unit. 


4? 


Condensed  Sumnmy  or  Set  Theony 


SYMdUMS) 

NAME 

KANINS 

VENN  DIAGRAM 

SEE 

PAGE 

u 

Univertel  S«t 

The  >ot  of  oil 

olononti  in  o 

glvon  ftudy. 

Shaded  port 

If  U. 

m 

u 

14 

• 

Null  Stt 

Tho  oot  contoininf 

no  oloaont* 

IS 

Subfot  Syfbol 

#,...i»  contained 

in../' 

Shaded  port 
la  oubtet. 

u 

11 

P 

Q 

Hull  of  Sou 

Any  capital  lotton 
noy  be  u*ed  to  ntae 
ooU.o.f.  A, 

o  o 

u 

4 

F 

Coapleaent 

Sot  of  oil  oloaonto 
In  U  not  in  P. 

Shaded  part 

u  r. 

g* 

fli 

u 

27 

C 

Eleaent 

Sy*ol 

"...io  on  oloaont 
of..." 

6 

i 

"...if  not  on 
oloaont  of..." 

6 

r  U  q 

union 

Sot  of  oil  oloaontt 
In  P  or  Q  or  both. 

Shaded  port 

U  P  U  Q 

u 

30 

p  n  q 

Intonoctlon 

Sot  of  oil  oloaonto 
cemoa  to  P  and  Q. 

Shaded  port 

it  p  r>  q. 

G© 

. 

u 

ss 

P  -  Q 

Difference 

Set  of  ail  ilnnu 
in  P  but  not  in  Q. 

Shaded  port 

it  p  -  q. 

u 

44 

POQ  •  0 

Dlojotnt  Sou 

Sou  which  hovo  no 
aeadtn  in  coaaon. 

Shaded  part  it 
PDq,  which  if 
capty  (there- 
fore  no  fhadfai 

O  G) 

u 

42 
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EXAMPLES  OF  DISPLAY  OPTIONS,  continued 


(continued) 

Example 


The  second  example  shows  the  same  information  presented  in 
prose  form  (only  half  r:  the  information  in  the  first  table 
is  given  in  this  example  in  older  to  save  space;  in  practice, 
of  course,  all  the  information  would  be  presented): 
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SIMULATED  COMPUTER  DISPLAY  UNIT 


Introduction 


Objectives 


Disgram 


In  order  to  have  a  glimpse  of  the  difficulties  and  possibil¬ 
ities  of  applying  information  mapping  to  computers  we  con¬ 
structed  a  simulated  computer  device. 

The  "computer"  is  operated  by  a  research  assistant  who  follows 
an  algorithm  that  tells  him  how  to  respond  to  different 
commands  given  by  the  user.  The  user  communicates  with  the 
"computer"  by  writing  commands  on  cards. 

The  research  assistant  finds  displays  manually  in  a  file  box, 
slips  them  into  the  dirplay  device  and  records  data  on  the 
interaction  such  as  time,  display  asked  for,  etc.  The  dis¬ 
plays  used  were  the  pages  of  the  information  map  book  on 
sets  and  probability. 


This  display  unit  was  built  primarily  as  a  tool  for  possible 
future  research  in  application  of  information  mapping  to  com¬ 
puters.  Thus,  we  were  primarily  concerned  with  the  following 
questions : 

•  Is  the  simulation  a  reasonable  one,  i.e.,  does  the  user 
actually  react  as  he  would  with  a  computer  based 
learning-reference  system? 

■'  How  complex  a  query  language  could  the  research  assis¬ 
tant  handle? 

•  Can  we  identify  specific  pages  that  cause  the  student 
trouble? 


The  last  question,  of  course,  is  concerned  with  revision 
of  the  book  pages.  Thus,  the  simulated  display  unit  served 
as  a  developmental  test  vehicle. 


Ill 


SIMULATED  COMPUTER  DISPLAY  UNIT,  continued 


Dimensions 

of 

Display 

Screen 


User 

Behavior 


Query 

Language 


Developmental 

Test 

Aspect 


NAME 

DEFINITION 

EXAMPLE 

QUESTION  1 

QUESTION  5 

ANSWER  1 

ANSWER  2 

17" -  » 


Only  a  few  students  were  run  on  this  device  so  a  more  defin¬ 
itive  answer  to  the  first  question  is  not  possible.  However, 
from  interviews  with  those  who  tried  it,  we  got  the  impression 
that  the  user  easily  entered  into  his  role,  became  forgetful 
of  the  research  assistant  behind  the  screen,  and  tended  to 
act  as  if  it  were  a  "real  machine." 


The  query  language  we  used  had  provisions  for  referencing 
a  table  of  contents  or  any  previous  page  or  topic.  In  addi¬ 
tion  the  user  was  able  to  determine  whether  or  not  he  wanted 
feedback  questions  or  review  questions. 

The  average  time  for  an  experienced  research  assistant  to 
respond  to  a  user  command  was  about  7  seconds.  We  found  this 
delay  to  be  acceptable  to  the  users  (although  we  hope  it  can 
be  reduced  still  more). 


The  results  of  the  data  on  time  per  page  gave  us  an  indica¬ 
tion  of  those  pages  that  gave  students  trouble.  Other 
indications  came  from  questions  asked  by  the  user  during  the 
session.  These  pages  were,  of  course,  revised. 

In  addition,  since  the  student  always  had  the  opportunity  to 
ask  for  feedback  questions  after  each  topic,  we  were  led  to 
insert  a  few  of  these  pages  where  we  previously  thought  them 
unnecessary. 


continued  on  next  page 
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SIMULATED  COMPUTER  DISPLAY  UNIT,  continued 


(continued) 

Developmental 

Test 

Aspect 


Some  time  data  for  two  subjects  who  learned  from  the  sets 
and  probability  units  are  shown  here: 


Time 

Fastest 

Slowest 

Factor 

Student 

Student 

Average  time  spent  per 

IM  page 

1.1  min 

2.5  min 

Range  of  time  spent  per 

IM  page 

0.25  -  2.50  min. 

1.00  -  6.33 
min. 

Average  time  spent  per 
feedback  question  page 

2.4  min 

2.8  min 

Range  of  time  spent  per 
feedback  question  page 

0.25  -  5.92  min. 

0.50  -  8.83 

min. 

Total  time  at  display 

2-1/6  hours 

4  hours 

Conclusion  The  simulation  approach  was  a  feasible  way  to  begin  investi¬ 
gating  properties  of  query  languages  and  different  types  of 
displays  ir  an  inexpensive  setting  without  the  long  waits 
associated  with  hardware  installation  and  software  preparation. 
We  found  it  was  possible  to  collect  meaningful  experimental 
data  on  various  display  research  issues  and  we  believe  that 
much  useful  work  can  be  done  with  such  simulated  units. 
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CHAPTER 


PRODUCTION  COSTS 


Introduction 


Cost 

Estimates 


Training 

the 

Writers 


During  the  project  we  wrote  a  total  of  approximately  twelve 
hours  of  material  on  sets,  probability,  and  permutations  and 
combinations.  Although  we  kept  records  of  the  amounts  of 
time  spent  on  the  different  aspects  involved  in  production, 
these  time  figures  are  confounded  with  the  time  we  spent 
defining  and  redefining  the  information  map  system.  Now 
that  the  system  has  been  established,  these  records  are  not 
representative  of  the  time  an  experienced  writer  would 
spend  at  the  task. 

In  order  to  obtain  a  more  realistic  estimate  the  authors 
tried  to  adjust  the  figures  to  reflect  what  might  be  involved 
in  the  production  of  an  information  map  book.  Thus,  the 
costs  we  present  are  estimates  based  on  the  consensus  of 
the  authors. 


The  total  cost  of  the  production  was  approximately  $12,000. 

As  indicated  in  the  evaluation  chapters,  the  combined  average 
time  to  study  the  book  was  about  twelve  hours  (note  that  no 
single  group  in  the  evaluative  studies  actually  went  through 
the  entire  book;  some  read  only  sets  and  probability,  while 
others  read  only  sets  and  permutations  and  combinations). 

We  estimated  that  the  breakdown  of  time  spent  on  the  different 
aspects  of  the  production  system  was  as  follows: 

Curriculum  planning  25% 

Writing  and  editing  55% 

Developmental  testing  20% 

Thus,  the  cost-per-hour  breakdown  is  as  follows: 

Curriculum  planning  $250 
Writing  and  editing  $550 

Developmental  testing  $200 

Total  $1000 


Since  we  have  developed  the  system,  we  have  no  data  on  the 
training  costs  which  might  be  involved.  However,  we  do 
have  experience  with  training  of  one  new  writer  who  came  on  the 
staff  after  the  preliminary  system  was  developed. 

The  indication  from  this  experience  points  to  the  need  for 
a  one  or  two  week  training  course  (reading  appropriate  research 


continued  on  next  page 


114 
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Training 

the 

Writers 


Comparison 
with  Pro¬ 
grammed  In¬ 
struction 


Comparison 
with  Compu¬ 
ter-Assisted 
Instruction 


and  actual  examples  of  information  map  books)  plus  a  month 
or  two  of  writing  experience  under  a  competent  editor.  This 
assumes  that  the  potential  writer  is  a  subject  matter  "expert" 
and  that  the  editor  is  well  versed  in  the  application  of 
learning  research  to  instructional  materials. 


Based  on  his  experience  as  Vice  President  of  Marketing  with 
Basic  Systems,  Inc.,  Murphy  (1967)  estimates  that  a  typical 
cost-per-hour  of  programmed  instruction  might  run  between 
$2500  and  $3000. 


Murphy's  figure  includes  direct  costs  plus  an  overhead  esti¬ 
mated  at  100%  of  direct.  Deleting  the  physical  production  costs , 
Murphy's  direct  cost-per-hour  runs  between- $11 25  and  $1375. 

Murphy's  figures  also  include  analysis  of  training  require¬ 
ments  which  involve  in  his  case  working  with  a  client.  We 
would  expect  that  our  curriculum  planning  involves  the 
same  type  of  work  and  so  is  more  or  less  comparable. 

Thus,  we  expect  that  information  mapping  costs  will  be 
approximately  the  same  as  those  of  programmed  instruction. 
However,  we  feel  that  we  have  produced  a  more  versatile 
product  (one  that  serves  both  initial  learning  and  reference 
uses) . 


Stolurow  (1969,  p.  308)  says  "It  is  difficult  to  determine 
the  cost,  but  one  approach  is  to  estimate  that  as  little  as 
100  hours  of  development  time  goes  into  one  hour  of  student 
time.  Estimates  up  to  400  hours  have  been  made,  but  defini¬ 
tive  data  are  not  available.  If  the  average  cost  per  hour 
for  the  writers  and  technical  personnel  is  estimated  at  $12, 
then  the  cost  would  vary  from  $1200  to  $4800  to  produce  one 
hour  of  student  instructional  material.  Other  costs  also 
are  involved,  but  hard  to  estimate.  They  include  key  punching, 
typing,  artwork,  machine  loading,  and  the  like." 

Carter  and  Walker  (1969,  p.333)  use  the  Suppes  figure  from 
Stanford  University  of  $5,000  per  student  hour  for  a  drill  and 
practice  mode  and  assume  a  $30,000  cost  per  hour  for  tutorial. 
They  do  usefully  point  out  that  spread  over  100,000  users, 
the  instructional  materials  development  costs  are  only  3  to 
6  percent  of  the  total  CAI  system. 
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Conclusions  Perhaps  the  only  concensus  that  can  be  reached  on  "average" 
costs  of  either  computer- assisted  instruction,  programmed 
instruction,  or  information  mapping  is  that  definitive  data 
are  not  available. 

Even  if  data  were  available,  however,  their  interpretation 
would  be  subject  to  a  great  deal  of  debate.  In  discussing 
decision-theory  approaches  to  the  problem  of  instructional 
research,  Luasdaine  (1963,  p.  667)  pointed  out 
"our  general  inability  to  make  reasonable  estimates  of  the 
costs,  and,  particularly,  the  anticipated  payoff  of 
educational  procedures." 
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Information  mapping  is  a  method  of  organizing  categories  of 
information  and  of  presenting  them  in  formats  that  communi¬ 
cate  quickly  with  the  user. 


Learning  and  reference  work  are  the  primary  applications  anti¬ 
cipated  for  the  system. 


The  procedures  and  techniques  of  information  mapping  are  de¬ 
rived  from  the  resources  of  the  education  world  and  incor¬ 
porate  effective  applications  in  display  and  communication 
technology. 


Information  mapped  materials  may  be  produced  in  book  form 
or  organized  into  data  bases  for  computer-aided  instruction. 

In  books  designed  for  initial  learning  and  reference,  the 
information  is  carried  in  clearly  labelled  information  blocks, 
arranged  in  an  order  prescribed  for  the  kind  of  information 
involved.  Other  features  of  these  self-instructional  books 
include  feedback  questions  and  answers,  special  maps  to 
facilitate  learning  and  retention,  charts  and  displays  for 
easy  retrieval  of  topics  for  review  and  reference  purposes. 

For  multi-purpose  computer  systems,  an  information  mapped  data 
base  would  be  composed  of  separable  labelled  blocks  of  infor¬ 
mation  together  with  their  interconnections.  Only  those 
parts  of  the  blocks  required  for  a  specific  purpose  need 
be  called  up.  This  flexible  system  would  permit  the  user  to 
organize  sequences  of  blocks  and  to  display  them  in  the  order 
that  best  serves  his  purpose,  whether  it  be  learning,  refer¬ 
ence,  or  browsing. 


Most  of  the  research  and  development  work  so  far  has  involved 
book  versions  of  topics  in  mathematics  and  computer  languages. 
The  primary  product  of  this  new  system  for  which  we  have  ex¬ 
perimental  data  is  a  self-instructional  book  on  sets  and 
probability  theory  for  college  students  with  minimal  prepara¬ 
tion  in  mathematics. 


One  of  the  main  tenets  of  information  mapping  is  that  the 
most  reliable  way  to  obtain  program  effectiveness  is  to  make 
empirical  testing  and  revision  an  integral  part  of  the  design 
and  development  process.  Thus  the  book  on  sets  and  probability 
was  shaped,  corrected  and  improved  by  tryout -and -revision 
cycles . 
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The  information  mapped  book  served  as  the  research  vehicle 
for  several  evaluative  studies  of  initial  learning  by  college 
students.  We  have  described  the  significant  effect  the  book 
had  on  the  students  and  we  have  presented  the  results  of 
attitude  studies  showing  that  students  find  the  materials 
attractive  and  valuable.  Whether  this  favorable  response 
is  colored  bv  the  novelty  of  the  approach  cannot  yec  be  deter¬ 
mined  foT  so  new  a  technique. 


Preliminary  spadework  was  carried  out  with  simulated  computer 
displays.  This  was  done  mainly  to  see  whether  the  simulation 
approach  was  feasible  for  exploring  whether  certain  options 
might  be  desirabl i  in  future  computer  applications:  query 
languages,  information- sequencing  rules,  dynamic  displays, 
and  so  on.  The  approach  seems  a  useful  and  economical 
way  of  researching  practical  issues  without  a  heavy  investment 
in  computer  facilities. 


Hie  production  of  learning  materials  in  college-level  mathe¬ 
matics  was  estimated  to  cost  about  the  same  as  programmed 
instruction.  However  the  versatility  with  which  information 
mapped  materials  can  be  adjusted  to  a  variety  of  purposes 
gives  it  obvious  cost  advantages  over  single-purpose  methods. 
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THE  INFORMATION  MAP  METHOD 


Introduction 


The  Logical 
Case 


We  said  earlier  that  our  research  related  primarily  to  the 
specific  product  tested,  the  sets  and  probability  book.  It 
would  be  a  logical  fallacy  to  interpret  the  results  as 
validating  a  method  if  this  be  conceived  of  as  the  whole 
class  of  products  that  might  arise  from  information  mapping 
techniques . 

Our  research  had  the  practical  objective  of  evaluating  the 
specific  program,  but  it  was  also  a  beginning  in  the  task 
of  learning  more  about  the  method  by  testing  hypotheses  about 
the  influence  of  certain  factors  and  variations  in  conditions. 

The  results  of  a  short  series  of  experiments  are  necessarily 
limited  in  their  generalizability,  but  we  consider  the  effects 
strong  enough  to  warrant  further  development  work. 


If  we  cannot  generalize  from  the  empirical  data  to  the 
"method,"  we  can  nevertheless  make  a  logical  case  for 
believing  that  the  method  is  a  promising  one.  In  the  first 
place,  it  draws  upon  the  resources  if  the  education  world, 
embodying  factors  for  which  there  is  already  support  in 
research  or  practice. 

Another  reason  for  expecting  the  products  to  be  effective  is 
that  they  are  developed  by  the  process  of  "formative  eval¬ 
uation,"  as  Scriven  (1967)  has  called  the  process  of  empirical 
testing  and  revision  throughout  development .  In  this  respect,  the 
information  map  product  has  the  advantage  of  a  steady  stream 
of  learner- response  data  from  all  areas  of  the  program.  These 
are  the  basis  for  program  improvement  so  that  a  desired 
level  of  proficiency  can  be  obtained  -  in  this  it  parallels 
programmed  instruction. 

Thus  since  the  effectiveness  of  given  products  can  be  engi¬ 
neered  by  practical  procedures,  there  is  little  point  to  ar¬ 
guing  whether  information  mapped  programs  are  as_  effective 
as  programs  produced  by  other  prescriptions.  The  information 
mapping  approach,  however,  has  certain  unique  advantages 
over  other  methods.  Some  of  these  are  indicated  on  the 
next  page. 


119 


SOME  ADVANTAGES  OF  INFORMATION  MAPPING 


Multi¬ 
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User 
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Retrieval 


Information  mapping  is  a  flexible  multi-purpose  system.  Its 
base  of  classified  information  blocks  permits  it  to  be 
used: 


for  book  production 

for  computer-based  programs 


The  separable  labelled  blocks  can  be  drawn  from  the  information 
base  as  needed  and  assembled  into  sequences  suitable  for: 

•  initial  learning 

•  for  the  naive  student 

•  for  the  experienced  student 

•  relearning  or  review 

•  reference  uses 

•  browsing  or  briefing 

Where  computer  scopes  are  available,  the  information  blocks 
may  be  presented  with  a  variety  of  display  options  such  as: 

•  static  versus  dynamic  displays 

•  whole  versus  sequential  display  of  parts 


The  block-identified  and  cross-indexed  information  base  can 
be  easily  modified;  therefore,  the  task  of  updating  the  changes 
in  a  system  is  relatively  simple. 


The  flexibility  of  the  system  would  permit  the  user  options 
that  are  not  available  with  other  methods.  The  user  may 
custom- tailor  his  own  program  by  calling  up  only  those  infor¬ 
mation  blocks  that  he  requires  for  his  specific  purpose. 


Programs  for  initial  learning  have  a  built-in  correction 
function:  practice  questions  and  answers  monitor  the  effects 
of  each  map.  Therefore  it  is  easy  to  identify  and  modify  the 
trouble  spots. 


Whether  in  book  version  or  in  computer-adapted  form,  the 
ease  with  which  information  can  be  retrieved  is  obvious:  in¬ 
formative  tables  of  contents,  maps  beginning  on  a  new  page, 
map  titles,  marginal  labels,  consistently  located  classes  of 
information,  related  page  numbers  at  end  of  maps,  charts, 
summary  tables,  indexes  --  all  are  designed  for  swift  commun¬ 
ication  with  the  user. 


continued  on  next  page 


120 


V 


SOME  ADVANTAGES  OF  INFORMATION  MAPPING,  continued 


Different  The  flexible  block  format  enables  one  to  adapt  each  learning 
Audiences  or  reference  program  to  the  special  interests  of  student 

groups.  For  instance,  the  worked  examples  and  feedback  ques¬ 
tions  of  our  probability  unit  were  geared  to  situations  of 
interest  to  students  of  the  behavioral  sciences.  The  same 
probability  text  can  be  modified  for  medical  students  or 
business  students  by  replacing  the  blocks  of  worked  examples 
and  the  pages  of  practice  questions. 


Research  Research  advantages  of  a  system  so  organized  are  manifold. 

Advantages  Since  the  content  characteristics  of  the  materials  can  be 
specified,  variations  in  the  make-up  of  learning  materials 
can  be  systematically  varied  with  relative  ease.  For  example, 
the  effects  of  varying  the  number  of  worked  examples  or  of 
changing  the  amount  of  redundant  information  could  be  explored. 

With  computer  capabilities,  many  questions  concerning  the 
optimal  complexity  of  displays  and  the  desirable  degree  of 
learner  control  over  instructional  sequencing  can  be  researched 
more  ecoromically  with  a  flexible,  modular  system. 


Comment  It  has  been  remarked  that  the  venerable  textbook  with  its  two 

distinctly  different  functions  as  both  learning  medium  and 
reference  source  may  soon  disappear  and  be  replaced  by  pro¬ 
grammed  instruction  plus  "the  well  designed  reference 
handbook"  (Lumsdaine,  1963,  p.  586). 

It  seems  that  a  better  solution  m  .*•  •  o  in  inf  ition  mapping. 
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APPENDIX 


MAP  CLASSIFICATION  CHART 


Types  of  Maps 

Description 

Information  Blocks 

Concepts 

A  concept  may  be  a 

•  technical  term 

•  generalization  sentence 

•  property  sentence 

•  rule  sentence 

•  relationship  sentence 

•  name  of  the  cor.cept 

•  definition  or 
description 

•  criteria 

•  generalization 

•  formula 

•  use  (purpose  or  function, 
worth,  value) 

•  example 

•  non-example 

•  introduction 

•  synonym 

•  notation 

•  diagram 

•  comment 
properties 

•  analogy 

Structures 

A  structure  is: 

•  a  physical  thing,  or 

•  something  which  can  be  di¬ 
vided  into  parts  which 
have  boundaries. 

•  name  of  structure 

•  meaning 

•  function 

•  /all  of  the  concept 
blocks/ 

•  parts  and  subparts 

•  boundaries 

•  diagram  (or  illustration, 
picture) 

Processes 

i 

A  process  is  some  structure 
changing  through  time.  The  des¬ 
cription  of  a  process  involves 
writing  about  what  happens  dur¬ 
ing  successive  stages  of  time. 

•  name  of  process 

•  /all  of  concept 
blocks/ 

•  /all  of  structure 
blocks/ 

•  purpose  of  process 

•  stage 

•  function  of  the  part 

•  cycle 

•  input 

•  esult  (output) 

•  occasion  for  starting 

•  changes 

•  time 

•  state 

continued  on  next  page 


MAP  CLASSIFICATION  CHART,  continued 


Types  of  Maps 

Description 

Information  Blocks 

(continued) 

Processes 

•  condition 

•  cause 

•  effect 

Procedures 

A  procedure  is  a  set  of  steps 
performed  to  obtain  some  speci¬ 
fied  outcome. 

•  name  of  procedure 

•  /all  of  concept 

blocks/ 

•  given 

•  step/procedure 

•  example 

•  when  to  use 

•  when  to  stop 

•  decisions 

Classifi¬ 

cations 

Classification  is  the  sorting  of 
things  by  concepts  into  cate¬ 
gories  by  the  use  of  one  or 
more  sorting  factors  (criteria) . 

•  name 

•  sorting  factor 

•  subclass  names 

•  purpose  of  sort 

Decisions 

Decision  tables  display  actions 
prescribed  for  different  con¬ 
ditions. 

•  name  of  decision 

•  "if”  parts  (or  condi¬ 

tions) 

•  "then"  parts  (or  ac¬ 

tions) 

example 

Facts 

Facts  are  sentences  containing 
arbitrary  associations  of  such 
things  as  syafcols,  measurements, 
dates  associated  with  events, 
experimental  results.  How  a 
fact  is  presented  depends  on 
the  context. 

statement  of  fact 

Proofs 

Proofs  3re  generally  used 
in  mathematical  subjects  for 
more  difficult  theorems. 

*  assumptions 
to  prove 

•  statement 

reason 

example 
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TEST-A  USED  IN  RESEARCH  REPORTED  IN  CHAPTER  S 


1.  Match  the  notation  on  the  left  with  the  correct  term  on  the  right. 


PfAOB) 

a. 

probability  of  the  complement  of  an  ev^nt 

S 

b. 

empty  event 

P(A) 

c. 

sample  space 

(AIB) 

d. 

conditional  probability  of  one  event  given 
another 

e. 

probability  of  the  intersection  of  two 
events 

f. 

independent  events 

2.  There  are  five  balls  in  an  urn.  Three  are  black  and  are  numbered  from 
one  to  three,  and  two  are  white  and  are  numbered  four  and  five.  We 
draw  a  ball  at  random  from  the  urn. 

A.  Assign  probabilities  to  the  element;.*:/  events.  _ 


B.  Find  the  probability  that  the  ball  drawn  had  an  odd  number  and 

was  black. _ 

C.  Find  the  probability  of  the  complement  of  the  event  in  question 

B.  _ 

D.  What  are  the  odds  in  favor  of  drawing  a  white  ball?  _ 


E.  What  is  the  conditional  probability  that  the  ball  is  black  given 
that  the  number  on  the  ball  is  odd?  _ 

3.  If  event  A  and  event  B  are  mutually  exclusive,  find  P  (AOB).  _ 


4.  We  have  two  urns.  Urn  I  contains  three  white  and  two  black  balls,  and 

Urn  II  contains  six  black  and  t..ree  green  balls. 

A.  If  we  choose  two  balls  at  random  from  Urn  I  without  replacement, 

what  is  the  probability  that  they  are  both  black?  _ • _ 

B.  If  we  choose  two  balls  at  random  from  Urn  II  with  replacement,  what 

is  the  probability  that  they  are  of  different  colors?  _ 

C.  If  we  choose  a  ball  at  random  from  the  first  urn  and  then  independ~ 

ently  choose  a  ball  at  random  from  the  second  urn,  what  is  the 
probability  that  neither  ball  is  black?  _ _ 


TEST- A,  continued 


g?v-? 


We  are  going  to  choose  two  balls  from  the  urns  in  the  following 
way:  Choose  the  first  ball  at  random  from  Urn  I.  If  this  ball  is 
white,  replace  it  and  choose  the  second  ball  at  random  from  Urn  I 
also.  If  the  first  ball  is  black,  however,  put  it  into  Urn  II  and 
choose  the  second  ball  at  random  from  Urn  II.  What  is  the  probabil¬ 
ity  that  both  balls  are  of  the  same  color?  _ 


5.  A  card  is  drawn  at  random  from  a  standard  deck.  What  is  the  probability 
that  the  card  is  a  heart  or  a  picture  card?  (Jack,  Queen  and  King  of 
each  suit  are  the  only  picture  cards.)  _  _ _ 


6.  If  A  and  B  are  independent,  and  P(A)  =  .8,  P(3)  =  .3,  find  the  probability 
that  A  or  B  occurs. 


A  coin  is  tossed  five  times,  or  until  a  head  appears,  whichever  comes 
first.  The  probability  of  a  head  on  any  toss  is  1/2  and  the  tosses  are 
independent. 


A.  List  the  points  of  the  sample  space. 


B.  Assign  probabilities  to  the  elementary  events. 


C.  What  is  the  probability  that  the  coin  is  tossed  less  than  five 

times?  _ _ _ _ 

D.  What  is  the  conditional  probability  that  the  coin  is  tossed  five 

times  given  that  the  first  three  tosses  all  result  in  tails?  _ 


8.  A  sample  space  contains  25  sample  points,  and  the  event  A  in  the  sample 
space  contains  10  points.  Find  the  probability  of  the  complement  of  A, 
assuming  all  the  outcomes  are  equally  likely.  _ 


9.  A  fair  coin  is  tossed  five  times.  Find  the  probability  of  getting  at 
least  one  head. 
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TEST- 8  USED  IN  RESEARCH  REPORTED  IN  CHAPTER  6 


1.  Match  the  notation  on  the  left  with  the  correct  term  on  the  right. 

a.  probability  of  an  elementary  event 

b.  empty  event 

c.  sample  spate 

d.  independent  events 

e.  probability  of  the  union  of  two  events 

f.  conditional  probability  of  one  event 
given  another 

2.  There  are  ten  balls  in  an  urn.  Three  are  white  and  are  numbered  from 
one  to  three,  and  seven  are  black  and  are  numbered  from  four  to  ten. 
We  draw  a  ball  at  random  from  the  urn. 

A.  Assign  probabilities  to  the  elementary  events.  _ 


B.  Find  the  probability  that  the  ball  drawn  had  an  odd  number  and 
was  black. 


C.  Find  the  probability  of  the  complement  of  the  event  in  B. 


D.  What  are  the  odds  in  favor  of  drawing  a  white  tall? 


E.  What  is  the  conditional  probability  that  the  ball  is  black  given 
that  the  number  on  the  Dali  is  odd? _ _ __ 

3.  If  event  A  and  event  B  arc  independent  and  each  has  probability  1/4, 

find  the  probability  of  their  intersection.  _ 

4.  We  have  two  urns.  Urn  I  contains  three  white  and  seven  black  balls, 
and  Urn  II  contains  two  black  and  two  green  balls. 

A.  If  we  choose  two’ balls  at  random  from  Urn  I  without  replacement, 

what  is  the  probability  that  they  are  both  black?  _ 

B.  If  we  choose  two  balls  at  random  from  Urn  II  with  replacement,  what 

is  the  probability  that  they  are  of  different  colors?  _ 

C.  If  we  choose  a  ball  at  random  from  the  first  urn  and  then  independ¬ 

ently  choose  a  ball  at  random  fxom  the  second  urn,  what  is  the 
probability  that  neither  ball  is  black?  _ 


0 

P  (BIA) 

P  ((0}) 

P  (AUB) 
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TEST-B,  continued 


D.  We  are  going  to  choose  two  balls  from  the  urns  in  the  following  way: 
Choose  the  first  ball  at  random  from  Urn  I.  If  this  ball  is  white, 
replace  it  and  choose  the  second  ball  at  random  from  Urn  I  also. 

If  tbe  first  ball  is  black,  however,  put  it  into  Urn  II  and  choose  the 
second  ball  at  random  from  Urn  II.  What  is  the  probability  that 
both  balls  are  of  the  same  color? _ _ 

5.  A  card  is  drawn  at  random  from  a  standard  deck.  What  is  the  probability 
that  the  card  is  a  spade  or  an  ace? 

6.  If  P(A)  =  .7,  P(B)  *  .2,  and  P(A|  B)  =  .5,  find  P(AOB). _ 

7.  Cards  are  drawn  at  random  from  a  standard  deck  until  a  black  card  appears, 
or  until  four  cards  have  been  drawn,  whichever  cqines  first.  Each  card 

is  replaced  before  the  next  card  is  drawn,  and  each  drawing  is  independent 
of  others. 

A.  List  the  points  of  the  sample  space.  _ _ 

B.  Assign  probabilities  to  the  elementary  events.  _ 


C.  What  is  the  probability  that  less  than  three  cards  are  drawn? 


D.  What  is  the  conditional  probability  that  four  cards  are  drawn  given 
that  the  first  card  is  red. 


8.  A  sample  space  contains  50  sample  points,  and  the  event  A  in  the  sample  1 

space  contains  5  points.  Find  the  probability  of  the  complement  of  A,  | 

assuming  all  outcomes  are  equally  likely.  _  f 

9.  A  fair  die  is  rolled  twice.  Find  the  probability  of  getting  at  least 
one  six. 
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